Conventional methods for determination of trace drug residues are either time consuming or labor intensive or require large specialized instruments, which hamper their practical applications in field analysis. Here, we present a rapid and quantitative surface-enhanced Raman scattering (SERS) detection method coupled with a portable Raman spectrometer for determination of trace drug residues on fish surface. Graphene oxide (GO) decorated popcorn-like Ag nanoparticles (NPs) on Cu plate (GO/AgNPs/Cu) were fabricated by a facile approach and directly employed as a robust SERS detection substrate. For practical SERS detections, trace-level residues of crystal violet (10−8 M, 4.1 ng/g) and malachite green (10−8 M, 3.6 ng/g) could be readily detected by simply swabbing the contaminated fish scale surface with the SERS substrate. Importantly, SERS detection was quantitatively realized in the broad linear concentrations. Compared with lab-based Raman spectrometer with large footprints, our method has potential applications in practical rapid, accurate, and on-site SERS determination.