The impacts of training image sizes and optimizers on deep convolutional neural networks for weed detection in alfalfa have not been well explored. In this research, AlexNet, GoogLeNet, VGGNet, and ResNet were trained with various sizes of input images, including 200 × 200, 400 × 400, 600 × 600, and 800 × 800 pixels, and deep learning optimizers including Adagrad, AdaDelta, Adaptive Moment Estimation (Adam) and Stochastic Gradient Descent (SGD). Increasing input image sizes reduced the classification accuracy of all neural networks. The neural networks trained with the input images of 200 × 200 pixels resulted in better classification accuracy than the other image sizes investigated here. The optimizers affected the performance of the neural networks for weed detection. AlexNet and GoogLeNet trained with AdaDelta and SGD outperformed Adagrad and Adam; VGGNet trained with AdaDelta outperformed Adagrad, Adam, and SGD; and ResNet trained with AdaDelta and Adagrad outperformed the Adam and SGD. When the neural networks were trained with the best-performed input image size (200 × 200 pixels) and the deep learning optimizer, VGGNet was the most effective neural network with high precision and recall values (≥0.99) in the validation and testing datasets. At the same time, ResNet was the least effective neural network for classifying images containing weeds. However, the detection accuracy did not differ between broadleaf and grass weeds for the different neural networks studied here. The developed neural networks can be used for scouting weed infestations in alfalfa and further integrated into the machine vision subsystem of smart sprayers for site-specific weed control.