How do product design teams converge on an idea?
We now have empirical evidence for the Double-Diamond model.
Is it possible to track, and in the future, guide, a design team's activities by analyzing their digital communication? That was the question I attempted to answer in the first publication of my PhD, with the specific aim of identifying convergence on a design idea. If you're really keen, you can read the entire open-access article on the Design Science website.
In this work, I used Slack messages from 32 product design teams as input into a specific type of topic modelling algorithm and used the statistics from these resulting models to represent the product design process. I found support for the Double-Diamond model — that design teams first diverge to ideate, converge on an idea, diverge to figure out how to execute the idea, and then converge on a final prototype. I present the communication qualities of successful design teams and a thematic analysis of the resulting topics. This work provides a better understanding of how we can optimize communication and collaboration in design teams. Most importantly, it highlights the benefits we can glean from studying the mass amounts of communication data that teams create daily.
Slack messages provide a rich and appealing dataset to study the design process.
Enterprise communication platforms, a fancy name for professional communication tools like Slack and Microsoft Teams, are a staple for student and industry teams across all disciplines, with recent statistics showing that 77% of Fortune 500 companies use Slack specifically. For designers, these tools lower the barrier to communication [1], allow synchronous and asynchronous collaboration [2], and help to build interpersonal relationships [3]. For researchers, these tools unlock big data that can be used to non-intrusively study designer processes using newly developed natural language processing (NLP) methods.
To assess how useful this data would be in studying designer processes, we chose an exploratory case study: modelling the convergence and divergence throughout the product design process. The product design process represents the sequential or iterative phases required when designing a product, such as the six-phase model proposed by Ulrich, Eppinger and Yang [4]. We were inspired by Dr. Andy Dong and colleagues’ use of natural language processing methods to measure coherent thinking [5] and knowledge construction [6] from designer communication.
A well-known model of the convergence and divergence in this process is the Double-Diamond model, shown below, which was popularized by The Design Council in 2005 [7]. This model states that design teams diverge to discover the problem, converge to define the area to focus on, diverge to develop potential solutions, and converge to deliver solutions that work. We attempted to match real data to this process using design teams' Slack data and NLP tools.
We collected data from 32 product design teams.
Before I detail the NLP method we chose, let's take a closer look at our data. This study analyzed over 250,000 Slack messages from design teams enrolled in four years' worth of a product design course — the fourth-year capstone course for MIT's mechanical engineers. These 17–20 people teams move through ideation to a functional prototype in a condensed design process, relying heavily on effective communication to coordinate work. This seven-phase design process, in which each phase is marked with a deliverable at the end, is approximately 93 days long. These teams frequently meet in person and supplement this face-to-face communication with Slack chat as their only digital communication platform. If you want to be thoroughly impressed (and feel a little bad about your own accomplishments at 22), I highly recommend checking out their final design showcase.
We used topic modelling to identify the number of topics discussed.
One way to estimate whether a design team is converging or diverging is to track how many different 'things', or topics, they are discussing. Luckily, a commonly used algorithm called topic modelling (an unsupervised machine learning method used to infer underlying topics from a set of documents) lets us do this [8]. We were motivated by the previous successful use of topic modelling to uncover insights in design contexts: Fu and colleagues’ measurement of example solution influence [9]; Gyory and team’s comparison of human and AI teams [10]; and Ball and peers’ analysis of capstone team performance [11]. Topic modelling algorithms return a set of topics, which are just lists of words and frequencies.
The most common topic modelling algorithm, Latent Dirichlet allocation (LDA), relies on a key assumption that each document contains many topics [8]. While this holds for the news articles and scientific manuscripts on which these algorithms are usually trained, it's unlikely that a single Slack message will contain multiple topics. To address this, researchers developed a version of this algorithm that relaxes the assumption to one topic per document, called short-text topic modelling, which has been used to analyze smaller documents such as tweets and news headlines [12]. In this work, we used the Gibs Sampling Dirichlet Mixture Model (GSDMM) algorithm [12], and while I won't go into the details of the algorithm here, you can read more in the short-text topic modelling tutorial linked below. We used this algorithm to model the topics in each team's communication in a specific phase, building a total of 224 individual topic models (32 teams x 7 design phases). A key feature of the GSDMM algorithm is that it converges on the optimal number of topics, unlike traditional topic modelling algorithms where the user has to input the estimated number of topics.
To observe convergence and divergence throughout the design process, we used two key statistics of these models: the optimal number of topics returned by the algorithm and the quality of the identified topics. A high-quality topic model will identify distinct and non-overlapping ideas which correlate highly with human judgement [13]. One automated measure of this quality is coherence, which measures how understandable a topic is [14]. There are many coherence metrics, and I recommend [13] and [14] for an overview.
We define convergent phases as having fewer topics which are of higher quality.
Let's connect these methods back to the purpose of the case study, to identify convergence and divergence in the product design process. We argue that the number of topics the model converged to in a given phase can be used as a measure of convergence; as teams converge on their product idea, they will discuss fewer topics. We also used topic coherence in combination with the number of topics as another indication of convergence: the more coherent the model is, the more confident we can be that the team is only discussing a set number of topics. With those defined, we can present our results.
The number of topics follows the proposed Double-Diamond pattern.
Looking first at the number of topics, we found a significant effect of the phase of the product design process on the number of topics. Focusing on figure b) below, we see that design teams discussed the most number of topics in the first phase, and this steadily decreased to a minimum in the Mockup Review and Final Selection phases, where the teams decide on a single product concept to pursue. The team diverges again in the Assembly Review and Technical Review phases, where they decide on the technical details of their product, and converge again when they present their final, functional prototype. When looking at the number of topics, we also found a significant main effect of team strength: teams rated as strong by the teaching staff discussed significantly fewer topics.
When we analyzed the quality of these models, we found that the models are most coherent in the Final Selection phase, when teams choose a final idea to pursue, and least coherent in the Sketch Model phase, where teams present six semi-fleshed out ideas. The quality of the models levels out in the last three phases as teams decide on how to execute their idea. When we consider the pattern in the number of topics together with the pattern in the quality of the topics, we can conclude that as teams converge on a design idea, their topic models have fewer topics, and these topics are of higher quality — which mimics the convergence in the Define stage of the Double-Diamond model. Then, in the second half of the design process, the teams diverge and converge — again mimicking the Double-Diamond model — but the quality of these models remains consistent. This consistency could be caused by a smaller vocabulary once the team has decided on a single idea, or it could represent the development of a 'shared voice' as team members become more comfortable with each other [15].
We found that teams overwhelmingly use Slack to make plans regarding designing, building, and testing products.
One of the benefits of topic modelling is its ability to provide rich results in the form of the words and frequencies that make up each topic. To utilize this rich data, we also looked for themes in the keywords for each topic. We built the categorization of these themes based on Wasiak and team’s taxonomy for classifying engineering emails [16]. We found an average of 4 themes present in each topic, and the most common theme by far, present in over 50% of topics, was the theme of planning. This finding tells us that the design teams studied may spend most of their time in-person having technical discussions and making design decisions, and they use Slack mostly for planning these activities.
Additionally, in divergent phases, we found that the teams talk more about the product — specifically its function, performance, features, manufacturing, and ergonomics — than in convergent phases. We found that teams use Slack to plan project deliverables more in the first half of the process but focus more on the development of the product near the end. Lastly, we found that, on average, stronger teams' topics contain more themes than weaker teams' topics, suggesting that strong teams may be more efficient in combining related themes into coherent discussions.
While we were able to model the design process using Slack messages, their short nature remains problematic.
The explosion of new communication technologies presents researchers with the opportunity to capture details that weren't available before and the challenge of dealing with this unstructured data. While we show that short-text topic modelling is useful in illuminating details of the design process and identifying some communication characteristics of successful design teams, we were faced with balancing a trade-off that affects the value of the topic modelling results. Short-text topic modelling requires many documents close to 100 words in length; since our Slack messages were around 16 words long on average, we combined messages sent to create a document. The more messages combined, the longer the document, but the fewer documents produced, and vice versa. While we obtained similar results for multiple document definitions, this trade-off will need to be managed in all topic modelling applications to short-chat messages.
Additionally, due to how often students discussed planning on Slack, we found that a deeper analysis of the topics was needed to truly make sense of the topic modelling results — the first few topic words were not always sufficiently descriptive, as seen below. While short-text topic modelling is more effective in this scenario than traditional topic modelling, we recommend combining this method with secondary analyses.
![A table showing two example topics, with columns representing Team ID, Team Strength, Phase, Topic Words, and the corresponding subthemes. The first row is a strong team in the Mockup Review phase, with topic words: [team name], compile, presentation, team, new, conference, room, pm, thank, grace, guy, tonight, great, quote, channel, beautiful, picture, side, usually, reminder. And subthemes: plans, project deliverables, and physical resources and tools.](https://miro.medium.com/v2/resize:fit:1400/format:webp/1*hF1LVNq1jG6yFGfdF8F75A.png)
Final Takeaways
In conclusion, we showed how we could use the communication artefacts that design teams produce daily to study designer processes and uncover the communication patterns of successful teams. We found that the number of topics discussed by teams follows the Double-Diamond pattern, strong teams discuss fewer topics than weak teams, and half of the topics identified contain evidence of planning. This work presents a step toward the possibility of using non-intrusive methods to track, guide, and assist a team's design process in real-time.
I want to thank all of the collaborators on this project: Kathy Cheng for her help with the qualitative data analysis; Georgia Van de Zande, David Wallace, and the entire 2.009 team for their support in data collection; and Alison Olechowski for her relentless problem-solving attitude and for never getting mad when I asked her to read it over just one more time.
If you found this interesting and would like to keep up with my work through the rest of my PhD, you can find me on Twitter (@_sharonferguson) and my website. Further, if you are part of an innovative, Slack-using team who would like to learn more about your collaboration and design processes and wouldn't mind allowing your data to be used to continue this research, please reach out! I can be reached at sharon.ferguson@mail.utoronto.ca.
References
[1] R. Brisco, R. I. Whitfield, and H. Grierson, "A novel systematic method to evaluate computer-supported collaborative design technologies," Research in Engineering Design, vol. 31, no. 1, pp. 53–81, 2020, doi: 10.1007/s00163–019–00323–7.
[2] J. A. Gopsill, H. C. McAlpine, and B. J. Hicks, "A Social Media framework to support Engineering Design Communication," Advanced Engineering Informatics, vol. 27, no. 4, pp. 580–597, Oct. 2013, doi: 10.1016/j.aei.2013.07.002.
[3] K. Abhari, N. Ascue, C. Boer, C. Sahoo, and M. Zarei, "Enterprise Social Network Applications: Enhancing and Driving Innovation Culture and Productivity Through Digital Technologies," presented at the Hawaii International Conference on System Sciences, 2021. doi: 10.24251/HICSS.2021.089.
[4] K. T. Ulrich, S. D. Eppinger, and M. C. Yang, Product Design and Development Seventh Edition. 2020.
[5] A. Dong, “Quantifying Coherent Thinking in Design: A Computational Linguistics Approach,” Design Computing and Cognition ’04, no. January, 2004, doi: 10.1007/978–1–4020–2393–4.
[6] A. Dong, “The latent semantic approach to studying design team communication,” Design Studies, vol. 26, no. 5, pp. 445–461, 2005, doi: 10.1016/j.destud.2004.10.003.
[7] Design Council, "Eleven lessons: managing design in eleven global brands A study of the design process," 2015.
[8] D. M. Blei, A. Y. Ng, and M. I. Jordan, "Latent Dirichlet allocation," Journal of Machine Learning Research, vol. 3, no. 4–5, pp. 993–1022, 2003, doi: 10.1016/b978–0–12–411519–4.00006–9.
[9] K. Fu, J. Cagan, and K. Kotovsky, “Design Team Convergence: The Influence of Example Solution Quality,” Journal of Mechanical Design, vol. 132, no. 11, p. 111005, Nov. 2010, doi: 10.1115/1.4002202.
[10] J. T. Gyory, B. Song, J. Cagan, and C. McComb, “Communication in Ai-Assisted Teams During an Interdisciplinary Drone Design Problem,” Proceedings of the Design Society, vol. 1, no. AUGUST, pp. 651–660, 2021, doi: 10.1017/pds.2021.65.
[11] Z. Ball, J. Bessette, and K. Lewis, “Who, What, and When? Exploring Student Focus in the Capstone Design Experience,” Proceedings of the ASME 2020 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference IDETC/CIE2020, pp. 1–12, 2020, doi: 10.1115/DETC2020-22027
[12] J. Yin and J. Wang, "A Dirichlet multinomial mixture model-based approach for short text clustering," Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 233–242, 2014, doi: 10.1145/2623330.2623715.
[13] K. Stevens, P. Kegelmeyer, D. Andrzejewski, and D. Buttler, "Exploring topic coherence over many models and many topics," EMNLP-CoNLL 2012–2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Proceedings of the Conference, no. July, pp. 952–961, 2012.
[14] M. Röder, A. Both, and A. Hinneburg, "Exploring the space of topic coherence measures," WSDM 2015 — Proceedings of the 8th ACM International Conference on Web Search and Data Mining, pp. 399–408, 2015, doi: 10.1145/2684822.2685324.
[15] A. Hill, S. Song, A. Dong, and A. Agogino, "Identifying shared understanding in design using document analysis," Proceedings of the ASME Design Engineering Technical Conference, vol. 4, no. October, pp. 309–315, 2001, doi: 10.1115/detc2001/dtm-21713.
[16] J. Wasiak, B. Hicks, L. Newnes, A. Dong, and L. Burrow, “Understanding engineering email: The development of a taxonomy for identifying and classifying engineering work,” Research in Engineering Design, vol. 21, no. 1, pp. 43–64, 2010, doi: 10.1007/s00163–009–0075–4.
