Hostname: page-component-848d4c4894-sjtt6 Total loading time: 0 Render date: 2024-06-24T22:09:00.408Z Has data issue: false hasContentIssue false
  • English
  • Français

Comprehensive assessment methods are key to progress in deep learning

Published online by Cambridge University Press:  06 December 2023

Michael W. Spratling
Michael W. Spratling*
Affiliation:
Department of Informatics, King's College London, London, UK michael.spratling@kcl.ac.uk https://nms.kcl.ac.uk/michael.spratling/
Get access Rights & Permissions [Opens in a new window]

Abstract

Bowers et al. eloquently describe issues with current deep neural network (DNN) models of vision, claiming that there are deficits both with the methods of assessment, and with the models themselves. I am in agreement with both these claims, but propose a different recipe to the one outlined in the target article for overcoming these issues.

Type
Open Peer Commentary
Information
Behavioral and Brain Sciences , Volume 46 , 2023 , e407
Check for updates
Copyright
Copyright © The Author(s), 2023. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Biggio, B., & Roli, F. (2018). Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 84, 317–331. doi:10.1016/j.patcog.2018.07.023CrossRefGoogle Scholar
Croce, F., & Hein, M. (2020). Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In H. Daumé III & A. Singh (Eds.), Proceedings of the international conference on machine learning, volume 119 of Proceedings of machine learning research (pp. 2206–2216). arXiv:2003.01690.Google Scholar
Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017). Neuroscience-inspired artificial intelligence. Neuron, 95, 245–258. doi:10.1016/j.neuron.2017.06.011CrossRefGoogle ScholarPubMed
Hendrycks, D., & Dietterich, T. G. (2019). Benchmarking neural network robustness to common corruptions and perturbations. In Proceedings of the international conference on learning representations, New Orleans, USA. arXiv:1903.12261.Google Scholar
Hendrycks, D., & Gimpel, K. (2017). A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of the international conference on Learning representations, Toulon, France. arXiv:1610.02136.Google Scholar
Johnson, M. H. (1999). Ontogenetic constraints on neural and behavioral plasticity: Evidence from imprinting and face recognition. Canadian Journal of Experimental Psychology, 53, 77–90.CrossRefGoogle Scholar
Kumano, S., Kera, H., & Yamasaki, T. (2022). Are DNNs fooled by extremely unrecognizable images? arXiv, arXiv:2012.03843.Google Scholar
Malhotra, G., Dujmović, M., & Bowers, J. S. (2022). Feature blindness: A challenge for understanding and modelling visual object recognition. PLoS Computational Biology, 18(5), e1009572. doi:10.1371/journal.pcbi.1009572CrossRefGoogle ScholarPubMed
Malhotra, G., Evans, B. D., & Bowers, J. S. (2020). Hiding a plane with a pixel: Examining shape-bias in CNNs and the benefit of building in biological constraints. Vision Research, 174, 57–68. doi:10.1016/j.visres.2020.04.013CrossRefGoogle ScholarPubMed
Michaelis, C., Mitzkus, B., Geirhos, R., Rusak, E., Bringmann, O., Ecker, A. S., … Brendel, W. (2019). Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv, arXiv:1907.07484.Google Scholar
Mu, N., & Gilmer, J. (2019). MNIST-C: A robustness benchmark for computer vision. arXiv, arXiv:1906.02337.Google Scholar
Nguyen, A., Yosinski, J., & Clune, J. (2015). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. arXiv, arXiv:1412.1897.Google Scholar
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., … Sutskever, I. (2021). Learning transferable visual models from natural language supervision. arXiv, arXiv:2103.00020. https://proceedings.mlr.press/v139/radford21a.htmlGoogle Scholar
Schrimpf, M., Kubilius, J., Lee, M. J., Murty, N. A. R., Ajemian, R., & DiCarlo, J. J. (2020). Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron, 108(3), 413–423 https://www.cell.com/neuron/fulltext/S0896-6273(20)30605-XCrossRefGoogle ScholarPubMed
Shen, Z., Liu, J., He, Y., Zhang, X., Xu, R., Yu, H., & Cui, P. (2021). Towards out-of-distribution generalization: A survey. arXiv, arXiv:2108.13624.Google Scholar
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. J., & Fergus, R. (2014). Intriguing properties of neural networks. In Proceedings of the international conference on learning representations, Banff, Canada. arXiv:1312.6199.Google Scholar
Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., & Madry, A. (2019). Robustness may be at odds with accuracy. In Proceedings of the international conference on learning representations, New Orleans, USA. arXiv:1805.12152.Google Scholar
Vaze, S., Han, K., Vedaldi, A., & Zisserman, A. (2022). Open-set recognition: A good closed-set classifier is all you need? In Proceedings of the international conference on learning representations, Virtual. arXiv:2110.06207.Google Scholar
Zaadnoordijk, L., Besold, T. R., & Cusack, R. (2022). Lessons from infant learning for unsupervised machine learning. Nature Machine Intelligence, 4, 510–520. doi:10.1038/s42256-022-00488-2CrossRefGoogle Scholar
Zador, A. M. (2019). A critique of pure learning and what artificial neural networks can learn from animal brains. Nature Communications, 10, 3770. doi:10.1038/s41467-019-11786-6CrossRefGoogle ScholarPubMed