Plasmids are autonomous extra-chromosomal elements in bacterial cells that can carry genes that are important for bacterial survival. There is considerable interest in the automated reconstruction of plasmid sequences from short-read whole genome sequence (WGS) data. To benchmark algorithms for automated plasmid sequence reconstruction, we selected 42 publicly available complete bacterial genome sequences with associated sequencing reads from 12 genera, containing 148 plasmids. We predicted plasmids from WGS with four different programs (PlasmidSPAdes, Recycler, cBar and PlasmidFinder) and compared the outcome to the reference sequences. Recall and precision were calculated to measure the completeness and accuracy of each prediction. PlasmidSPAdes reconstructs plasmids based on coverage differences in the assembly graph. It reconstructed most of the reference plasmids (recall = 0.82) with approximately a quarter of the predicted sequences corresponding to false positives (precision = 0.76). A total of 83.1 % of the reconstructions from genomes with multiple plasmids were merged and manual steps were necessary to separate individual plasmid sequences. Recycler searches the assembly graph for sub-graphs corresponding to circular sequences. It correctly predicted small plasmids but failed with long plasmids (recall = 0.12, precision = 0.28). cBar, which applies pentamer frequency composition analysis to detect plasmid-derived contigs, showed an overall recall and precision of 0.77 and 0.63. However, cBar only categorizes contigs as plasmid-derived and does not bin the different plasmids correctly within a bacterial isolate. PlasmidFinder, which searches for matches in a replicon database, had the highest precision (1.0) but was restricted by the contents of its database and the contig length obtained from de novo assembly (recall = 0.33). Based on this analysis we conclude that without long read information, plasmid reconstruction from WGS remains challenging and error-prone.