In Depth Response to Shipstead et al. (2012)
For a pdf version of this response, click here
To read the Cogmed Claims and Evidence, click here
Based on more than 25 peer-reviewed studies, Cogmed is the most evidence-based computer intervention for working memory (WM) and attention in the world. No other commercial WM training program so actively pursues research validation or employs such discretion in aligning claims with evidence.
Contrary to the recent assertions by Shipstead et al. (2012), trainees consistently exhibit improved WM capacity on non-trained simple (Brehmer et al., 2012, Klingberg et al., 2002; 2005, Thorell et al., 2009) and complex span tasks immediately (Holmes et al., 2009; 2010, Bergman Nutley et al., 2011) and up to six months after Cogmed (Holmes et al., 2009), including increased ability to follow instructions (Holmes et al., 2009) and to remember and add digits (Westerberg et al. 2007). Further, there are randomized, placebo controlled investigations from three separate research groups that demonstrate improved attention in everyday life post-Cogmed (Klingberg et al., 2005; Brehmer et al., 2012; Green et al., 2012). Even as the reviewers object to the use of non-adaptive control groups in being subjectively different and less supportive, studies using Cogmed have measured motivation in both adaptive and non-adaptive (placebo) groups and found them to be equal (Bergman Nutley et al., 2011) as well as, included active control groups with both non-adaptive training (Bergman Nutley et al., 2011; Green et al., 2012) and computer games (Thorell et al., 2009) to demonstrate improvements after Cogmed training.
Before addressing any further specific objections, let us be clear about what Cogmed is in fact claiming:
(1) WM is key to attention and learning
(2) WM can be improved by training, using right tool and protocol: Cogmed
(3) WM can be improved by training in all age ranges
(4) Training-related improvement can be shown on three levels of assessment: fMRI/PET, neuropsychological testing and rating scales
(5) Improved WM generalizes to improvements in daily functioning
(6)Improvements in WM and behavioral outcomes are sustained at 6 months post training
(7) Effects of WM training are specific: WM and its derived functions are improved, but there is no across-the-board-improvement
(8) Training effects have substantial real world impact on individuals impaired by their WM capacity.
In addition to the peer-reviewed research, clinical data from almost a decade of real world implementation provides Cogmed the confidence to stand behind these claims. Note also that Cogmed never asserts that all of the questions regarding the underlying mechanisms or efficacy of WM training have been answered. Rather, Cogmed openly acknowledges that crucial questions remain and thus supports over 60 ongoing research projects by independent investigators so to foster greater understanding of the impact of WM training. Our charge is to neither become embroiled in a theoretical debate nor discouraged by unduly negative critique, but instead to move the conversation forward with better investigative designs and through application of research findings to the continued evolution of the field.
In evaluating the review by Shipstead et al. (2012), it seems useful to approach the core sections of the paper, providing remarks from Cogmed as well as sharing insights from researchers whose commentaries were peer-reviewed and published alongside the Shipstead article.
Does Cogmed increase WM capacity?
In their review, Shipstead et al. (2012) question whether Cogmed improves WM capacity, reasoning that simple span tasks are not adequate for assessment of near transfer to WM. However, based on the extant literature it should be argued that evidence for improved WM capacity and attention after Cogmed is, in fact, strong. Independent researchers providing comment on the review stated:
“…Our interpretation of the reviewed studies is that there are indeed “near transfer” effects, that is, improvements in tasks that are within the trained domain (WM skills and attentional processes)…” (Jaeggi et al., 2012).
This observation is based on more than 20 studies showing that Cogmed has improved performance on non-trained WM tasks, including not only simple span tasks, but also complex span tasks (Holmes et al., 2009, 2010; Bergman Nutley et al., 2011). Participants have improved significantly on WM tasks in a non-trained modality (ie., training on only visuo-spatial WM tasks transferred to improvement on non-trained verbal WM tasks), non-trained transfer tasks such as remembering and executing instructions (Holmes et al., 2009), and remembering and adding digits (PASAT; Westerberg et al., 2007; Brehmer et al., 2012). Stemming from this body of evidence, Jaeggi et al. (2012) held:
“…we do not agree with the authors’ evaluation that simple span tasks such as digit span should be avoided to assess WM processes. First of all, simple span tasks have been useful to predict academic achievement (see e.g. Gathercole, Pickering, Knight, & Stegman, 2004), as well as ADHD symptoms (McInnes, Humphries, Hogg-Johnson, & Tannock, 2003), and there is considerable literature demonstrating that STM and WM tasks share considerable variance…Ironically, to suggest not using STM tasks because of a lack of reliability would also question quite a few of the studies coming from the authors’ own laboratory using simple span tasks in their individual difference research (e.g. Engle, Tuholski, Laughlin, & Conway, 1999; Kane et al., 2004)” (Jaeggi et al., 2012).
It is thus important to clarify that near transfer has been demonstrated extensively in the Cogmed literature through use of WM tasks that differ in presentation and response mode from the trained tasks.
Does Cogmed improve reasoning ability?
Unfortunately, Shipstead et al. (2012) misrepresents the claims made by Cogmed. For example, the authors devote a great deal of effort to demonstrating that the Cogmed “claim” to improve reasoning abilities is unfounded. Although the authors may legitimately argue that researchers should use a variety of tasks to assess reasoning before and after training, they may not legitimately state that Cogmed claims are “in this area relatively subdued”. In fact, the Cogmed claims in this area are non-existent. Although some of the Cogmed research has found improved reasoning after training, the results have not yet been consistent so to warrant an official claim.
Does Cogmed train attention?
The reviewers assert that Cogmed has evidence for improved “attentional stamina” but inconsistent results on the Stroop task, a measure framed as a better test of attention. Although the Stroop task may be an important measure, it is by no means the best or only measure of attention and arguably a test of inhibitory control as opposed to top-down attention. Based on neuro-imaging data, the neural basis of the training effect in Cogmed is a common network of brain regions in the intraparietal and prefrontal cortex which underlies some WM demanding tasks as well as top-down attention. It is therefore not surprising that findings regarding the Stroop task are less consistent than the strong effects observed on attention tasks. As put forth in Klingberg et al. (2012):
“…there is solid evidence that Cogmed working memory training improves working memory capacity as well as top down attention, measured both with cognitive tasks and estimates of attention in everyday life. Changes in both working memory and attentive behavior are relevant for children in the classroom situation independent of any theoretical disputes about the true nature of working memory capacity” (Klingberg, 2012).
In summary, there are at least three randomized, placebo-controlled investigations demonstrating improved attention in everyday life after (Klingberg et al., 2005; Brehmer et al., 2012; Green et al., 2012) and various studies with less stringent designs (Beck et al., 2010, Mezzacappa & Buckner, 2010) showing improvements in inattentive symptoms.
Does Cogmed alleviate ADHD-related symptoms?
In regards to ADHD-related symptoms, Shipstead et al. (2012) asserts that any claim that Cogmed alleviates ADHD symptoms is unwarranted. Importantly, Cogmed does not make any specific claims with regards to the diagnostic category of ‘ADHD’. Rather, research evidence supports the use of Cogmed to improve attention, a particular symptom within the cluster of symptoms that comprise an ADHD diagnosis.
That said, various studies note that children with ADHD show a deficit in ‘attentional stamina’, also known as ‘sustained attention’ (Muir-Broaddus et al., 2002). Also, studies have linked ‘sustained attention’ to academic achievement (Hunt & Randhawa, 1983). Similarly, ‘vigilance’ is another term commonly used to capture the concept of ‘sustained attention’. Vigilance has often been linked to academic achievement differentiating high achieving students from low achieving students (Krichner & Knopf, 1974; Hatta, 1993; Hatta et al., 1996; Bauermeister et al., 2005). In adulthood, those with ADHD show a deficit in vigilance as well as significantly more academic problems in school (Seidman et al., 1998). Finally, a meta-analysis by Wilcutt et al. (2005) noted that ‘vigilance’ was one of the variables with the strongest and most consistent effect size that distinguished ADHD groups from non-ADHD groups.
Given the preponderance of data linking poor ‘attentional stamina’, ‘sustained attention’ and/or ‘vigilance’ to ADHD, an improvement in this area strikes at the core of the problems of ADHD. Data is easily found which links a deficit in ‘attentional stamina’ to poorer academic functioning.
Additionally, the neuro-scientific data suggests that there are overlaps in the brain where the functions of ‘sustained attention’, selective attention and ‘divided attention’ originate. Finally, for those with ADHD problems, both ‘attentional stamina’ and academic achievement persist into adulthood.
Importantly, research with ADHD children has shown increased attention on DSM-IV criteria for inattention after Cogmed (Klingberg et al., 2005). Typically functioning older and younger adults have also significantly improved on their inattentive behavior after Cogmed (Brehmer et al., 2012). Based on this evidence, Cogmed can confidently claim to improve attention, a known symptom in ADHD.
To provide further example, Green et al. (2012) studied a sample of 26 children, ages 7 to 14 years, diagnosed with ADHD (combined or inattentive type) in a randomized, placebo controlled Cogmed Working Memory Training trial. Prior to and post-training, children in both groups were assessed using the Restricted Academic Situations Task (RAST), an observational system used to assess aspects of off-task behavior during the completion of an academic task. At the start of the task, children were provided a toy or game of their choice to play with and after 5 minutes, the researcher re-entered the room and moved the game to the side while telling the child to complete a set of academic worksheets for 15 minutes. Before the researcher left the room, they instructed the child not to leave their seat or play with any toys.
The researcher then observed through a one way mirror and coded the occurrences of five behaviors during the academic task: off task (looks away from the paper), out of seat (leaves chair), fidgets (repetitive purposeless motion), vocalizes, and plays with objects (touches any object in the room unrelated to the task). Importantly, researchers were blind as to whether a child received adaptive or placebo Cogmed training. At post-test, children in the adaptive training condition demonstrated significantly improved WM on the Working Memory Index of the WISC-IV as well as, decreased behaviors in the off task category and the plays with objects category. These findings imply that the influence of training had a significant effect on inattentive behaviors that are frequently associated with ADHD and which are related to academic functioning. Thus, Green et al. (2012) demonstrated that children in adaptive Cogmed training improve not only on standardized assessments of WM but also, on an ecologically valid measure of observable ADHD–associated behaviors that would substantially impact their functioning in the real world. Thus, as noted in Shipstead et al. (2012): “There is evidence that Cogmed training will improve ‘attentional stamina’…” and based on the existing literature, this places Cogmed as an intervention that addresses a core symptom experienced in ADHD.
It is in recognizing the key insights and in refuting some the conclusions of Shipstead et al. (2012) that the importance for a pragmatic approach for evaluating cognitive intervention research comes to light. The path to gold standard studies is long, expensive, and dependent on foundational studies that lay the groundwork for larger, more elegant designs. A rather practical approach to the Shipstead et al. (2012) review was put forth by Gathercole et al. (2012) as they explain the importance of maintaining a critical eye but also, of giving due value to each study contributing the body of research:
“We have found generalized working memory enhancements in low-memory children (Dunning, Holmes, & Gathercole, undergoing revision), upholding our previous findings using less stringent designs. Without the preliminary evidence that the intervention could work, the full RCT study would not have been justified. However, the predictive value of such data is lost if every study that fails to meet the gold standard criteria is relentlessly rejected” (Gathercole et al., 2012).
It is therefore important to grasp that no one study or even collection of studies will answer every question or fulfill all design criteria. The reviewers themselves recognized the constraints of time, funding, and increased risk of confounding variables when they recently, and it seems for the first time, conducted their own n-back WM training study (Redick et al., 2012). While balancing skepticism and practicality, review of the literature should also reflect the objectivity and equanimity of the researchers who commented on the review. As reflected in the titles of the commentaries (e.g., “The future promise of Cogmed working memory training”, “Cogmed training: Let’s be realistic about intervention research”), these researchers provided a much less cynical view of the state of WM training research. Shah et al. (2012) explained:
“…research on Cogmed may not be airtight, but the Shipstead et al. article is overly pessimistic. Our view is consistent with the guidelines for “empirically supported therapies” developed by a task force of clinical scientists (Chambless & Ollendisk, 2001). Cogmed meets the criteria for “promising interventions” (p. 689, Table 1)” (Shah et al., 2012).
Regarding Cogmed’s commitment to the primacy of research, in stark contrast to other commercial providers of cognitive training products, Morrison & Chein (2012) remarked:
“Still, the investigators who have worked on the development of this product should be acknowledged for their honest and scientifically rigorous exploration of its potential. The preliminary findings were not overstated, and have been followed-up in a sizable number of subsequent studies... In published research, those presenting evidence supporting the efficacy of Cogmed have typically demonstrated restraint in not over-interpreting the data, and have been forthcoming in instances where outcomes did not meet expectations.” (Morrison & Chein, 2012).
In closing, Cogmed agrees that a healthy dose of skepticism in research is good for science. However, it is too soon to allow pessimism to dominate the conversation. To answer the question posed in the headline by the reviewers: Cogmed Working Memory Training is effective and the evidence supports the claims. Hundreds of clinicians and researchers are looking forward, understanding what is known about WM training, and questioning what else can be learned. Future directions involve investigations into the cognitive profiles and developmental stages that are most appropriate for WM training, experimentation with tasks are most effective for training, and determination of which training regimens are best. As the evidence base for Cogmed grows, it is expected that the findings will be evaluated and challenged. It can only be hoped that reviewers will fairly approach this discourse in the future.
Cogmed Research References
Beck, S.J., Hanson, C.A., Puffenberger, S.S., Benninger, K.L., & Benninger, W.B. (2010).A controlled trial of working memory training for children and adolescents with ADHD. Journal of Clinical Child and Adolescent Psychology, 39(6), 825 -836. Doi: 10.1080/15374416.2010.517162
Bellander, M., Brehmer, Y., Westerberg, H., Karlsson, S., Fürth, D., Bergman, O., Eriksson, E., & Bäckman, L. (2011). Preliminary evidence that allelic variation in the LMX1A gene influences training-related working memory improvement. Neuropsychologia, 49, 1938 -1942.doi:10.1016/j.neuropsychologia.2011.03.021
Brehmer, Y., Rieckmann, A., Bellander, M., Westerberg, H., Fischer, H., & Bäckman, L. (2011). Neural correlates of training-related working-memory gains in old age. Neuroimage, 58(4),1110-1120.doi:10.1016/j.neuroimage.2011.06.079
Brehmer, Y., Westerberg, H., & Bäckman, L. (2012). Working-memory training in younger and older adults: Training gains, transfer, and maintenance. Frontiers in Human Neuroscience, 6(63), 1-7.doi:10.3389/fnhum.2012.00063
Brehmer, Y., Westerberg, H., Bellander, M., Fürth, D., Karlsson, S., & Bäckman, L. (2009). Working memory plasticity modulated by dopamine transporter genotype. Neuroscience Letters, 467, 117 -120.doi:10.1016/j.neulet.2009.10.018
Bergman-Nutley, S., Söderqvist, S., Bryde, S., Thorell, L.B., Humphreys, K., & Klingberg, T. (2011). Gains in fluid intelligence after training non-verbal reasoning in 4-year-old children: A controlled, randomized study. Developmental Science, 14(3), 591 -601. doi:10.1111/j.1467-7687.2010.01022.x
Dahlin, K. (2011). Effects of working memory training on reading in children with special needs. Reading and Writing, 24, 479-491.doi:10.1007/s11145-010-9238-y
Diamond, A., & Lee, K. (2011). Interventions shown to aid executive function development in children 4 to 12 years old [Special section]. Science, 333, 959-963. doi: 10.1126/science.1204529
Gibson, B.S., Gondoli, D.M., Johnson, A.C., Steeger, C.M., Dobrzenski, B.A., & Morrissey, R.A.(2011). Component analysis of verbal versus spatial working memory training in adolescents with ADHD: A randomized, controlled trial. Child Neuropsychology, 17(6), 546-563.doi:10.1080/09297049.2010.551186
Gibson, B.S., Kronenberger, W.G., Gondoli, D.M., Johnson, A.C., Morrissey, R.A., & Steeger, C.M. (2012). Component analysis of simple span vs. Complex span adative working memory exercises: A randomized, controlled trial. Journal of Applied Research in Memory and Cognition. In Press.
Green, C.T., Long, D.L., Green, D., Iosif, A., Dixon, F., Miller, M.R., Fassbender, C., & Schweitzer, J.B. (2012). Will working memory training generalize to improve off-task behavior in children with Attention-Deficit/Hyperactivity Disorder? Neurotherapeutics. Advance online publication.doi:10.1007/s13311-012-0124-y
Holmes, J., Gathercole, S.E., & Dunning, D.L. (2009). Adaptive training leads to sustained enhancement of poor working memory in children. Developmental Science, 12(4), F9 -F15. doi: 10.1111/j.1467-7687.2009.00848x
Holmes, J., Gathercole, S.E., & Dunning, D.L. (2010). Poor wokring memory: Impact and interventions. In J. Holmes (Ed.), Advances in Child Development and Behavior Developmental Disorders and Interventions, Volume 39 (pp. 1- 43). Burlington: Academic Press.
Holmes, J., Gathercole, S.E., Place, M., Dunning, D.L., Hilton, K.A., & Elliot, J.G. (2010). Working memory deficits can be overcome: Impacts of training and medication on working memory in children with ADHD. Applied Cognitive Psychology, 24(6), 827-836. doi: 10.1002/acp.1589
Johansson, B., & Tornmalm, M. (2012). Working memory training for patients with acquired Brain injury: Effects in daily life. Scandinavian Journal of Occupational Therapy, 19(2), 176-183. doi:10.3109/11038128.2011.603352
Klingberg, T. (2010). Training and plasticity of working memory. Trends in Cognitive Sciences, 14(7), 317 -324. doi: 10.1016/j.tics.2010.05.002
Klingberg, T., Fernell, E., Olesen, P.J., Johnson, M., Gustafsson, P., Dahlström, K., Gillberg, C.G.,
Forssberg, H., & Westerberg, H. (2005). Computerized training of working memory in children with ADHD – a randomized, controlled trial. Journal of the American Academy of Child & Adolescent Psychiatry, 44(2), 177-186.
Klingberg, T., Forssberg, H., & Westerberg, H. (2002). Training of working memory in children with ADHD. Journal of Clinical and Experimental Neuropsychology, 24(6), 781 -791.
Kronenberger, W.G., Pisoni, D.B., & Henning, S.C., & Colson, B.G., & Hazzard, L.M. (2011). Working memory training for children with cochlear implants: A pilot study. Journal of Speech, Language, and Hearing Research, 54(4), 1182 -1196.
Løhaugen, G.C.C., Antonsen, I., Håberg, A., Gramstad, A., Vik, T., Brubakk, A.M., & Skranes, J. (2011). Computerized working memory training improves function in adolescents born at extremely low birth weight. Journal of Pediatrics, 158(4), 555- 561.
Lundqvist, A., Gundström, K., & Rönnberg, J.(2010). Computerized working memory training in a group of patients suffering from acquired brain injury. Brain Injury, 24(10), 1173- 1183.
Mcnab, F., Varrone, A., Farde, L., Jucaite, A., Bystritsky, P., Forssberg, H., & Klingberg, T. (2009). Changes in cortical dopamine D1 receptor binding associated with cognitive training. Science, 323, 800 - 802.doi:10.1126/science.1166102
Mezzacappa, E. & Buckner, J.C. (2010). Working memory training for children with attention problems or hyperactivity: A school-based pilot study. School Mental Health, 2(4), 202- 208.doi: 10.1007/s12310-010-9030-9.
Olesen, P.J., Westerberg, H., & Klingberg, T. (2004). Increased prefrontal and parietal activity after training of working memory. Nature Neuroscience, 7(1), 75- 79. Doi:10.1038/nn1165
Roughan, L., & Hadwin, J.A. (2011). The impact of working memory training in young people with social, emotional and behavioral difficulties. Learning and Individual Differences, 21, 759-764.doi:10.1016/j.lindif.2011.07.011
Söderqvist, S., Nutley, S.B., Ottersen, J., Grill, K.M., Klingberg, T. (2012). Computerized training of non-verbal reasoning and working memory in children with intellectual disability. Frontiers in Human Neuroscience, 6, 271. doi: 10.3389/fnhum.2012.00271
Söderqvist, S., Bergman Nutley, S., Peyrard-Janvid, M., Matsoon, H., Humphreys, K., Kere, J., Klingberg, T. (2012). Dopamine, working memory, and training induced plasticity: Implications for developmental research. Developmental Psychology, 48(3), 836-843. Doi:10.1037/a0026179
Thorell, L.B., Lindqvist, S., Bergman Nutley, S.,Bohlin, G. & Klingberg, T. (2009). Training and transfer effects of executive functions in preschool children. Developmental Science, 12(1), 106 -133.doi:10.1111/j.1467-7687.2008.00745
Westerberg, H., & Klingberg, T. (2007). Changes in cortical activity after training of working memory – a single-subject analysis. Physiology and Behavior, 92(1-2), 186 -192. Doi:10.1016/j.physbeh.2007.05.041
Westerberg, H., Jacobaeus, H., Hirvikoski, T., Clevberger, P., Östensson, M.L., Bartfai, A., & Klingberg, T. (2007). Computerized working memory training after stroke – a pilot study. Brain Injury, 21(1), 21-29.doi:10.1080/02699050601148726
Researcher Commentary References
Gathercole, S.E., Dunning, D.L., & Holmes, J. (2012). Cogmed training: Let’s be realistic about intervention research. Journal of Applied Research in Memory and Cognition, 1(3), 201-203. http://dx.doi.org/10.1016/j.jarmac.2012.06.003
Gibson, B.S., Gondoli, D.M., Johnson, A.C., Steeger, C.M., & Morrissey, R.A. (2012). The future promise of Cogmed working memory training. Journal of Applied Research in Memory and Cognition, 1(3),214-216. http://dx.doi.org/10.1016/j.jarmac.2012.06.003
Hulme, C. & Melby –Levårg, M. (2012). Current evidence does not support the claims made for CogMed working memory training. Journal of Applied Research in Memory and Cognition, 1(3),197-200. http://dx.doi.org/10.1016/j.jarmac.2012.06.003
Jaeggi, S.M., Buschkuehl, M., Jonides, J., & Shah, P. (2012). Cogmed and working memory training –Current challenges and the search for underlying mechanisms. Journal of Applied Research in Memory and Cognition, 1(3), 211-213. http://dx.doi.org/10.1016/j.jarmac.2012.06.003
Klingberg, T. (2012). Is working memory capacity fixed? Journal of Applied Research in Memory and Cognition, 1(3),194-196. http://dx.doi.org/10.1016/j.jarmac.2012.06.003
Logie, R.H. (2012). Cognitive training: Strategies and the multicomponent cognitive system. Journal of Applied Research in Memory and Cognition, 1(3), 206-207. http://dx.doi.org/10.1016/j.jarmac.2012.06.003
Morrison, A.B., & Chein, J.M. (2012). The controversy over Cogmed. Journal of Applied Research in Memory and Cognition, 1(3), 208-210. http://dx.doi.org/10.1016/j.jarmac.2012.06.003
Shah, P., Buschkuehl, M., Jaeggi, S., & Jonides, J. (2012). Cognitive training for ADHD: The importance of individual differences. Journal of Applied Research in Memory and Cognition, 1(3), 204-205. http://dx.doi.org/10.1016/j.jarmac.2012.06.003
Shipstead, Z., Hicks, K.L., & Engle, R.W. (2012). Working memory training remains a work in progress. Journal of Applied Research in Memory and Cognition, 1(3), 217 -219. http://dx.doi.org/10.1016/j.jarmac.2012.06.003
Review Article References
Shipstead, Z., Hicks, K.L., & Engle. R.W. (2012). Cogmed Working Memory Training: Does the evidence support the claims? Journal of Applied Research in Memory and Cognition. In Press.
Shipstead, Z., Redick, T.S., & Engle, R.W. (2010). Does working memory training generalize? Psychologica Belgica, 50(3 & 4), 256 -276.
Shipstead, Z., Redick, T.S., & Engle, R.W. (2012). Is working memory training effective? Psychological Bulletin. doi:10.1037/a0027473