PURPOSE: Rule-based approaches to determining glaucoma progression from visual fields (VFs) alone are discordant and have tradeoffs. To detect better when glaucoma progression is occurring, we used a longitudinal data set of merged VF and clinical data to assess the performance of a convolutional long short-term memory (LSTM) neural network. DESIGN: Retrospective analysis of longitudinal clinical and VF data. PARTICIPANTS: From 2 initial datasets of 672 123 VF results from 213 254 eyes and 350 437 samples of clinical data, persons at the intersection of both datasets with 4 or more VF results and corresponding baseline clinical data (cup-to-disc ratio, central corneal thickness, and intraocular pressure) were included. After exclusion criteria-specifically the removal of VFs with high false-positive and false-negative rates and entries with missing data-were applied to ensure reliable data, 11 242 eyes remained. METHODS: Three commonly used glaucoma progression algorithms (VF index slope, mean deviation slope, and pointwise linear regression) were used to define eyes as stable or progressing. Two machine learning models, one exclusively trained on VF data and another trained on both VF and clinical data, were tested. MAIN OUTCOME MEASURES: Area under the receiver operating characteristic curve (AUC) and area under the precision-recall curve (AUPRC) calculated on a held-out test set and mean accuracies from threefold cross-validation were used to compare the performance of the machine learning models. RESULTS: The convolutional LSTM network demonstrated 91% to 93% accuracy with respect to the different conventional glaucoma progression algorithms given 4 consecutive VF results for each participant. The model that was trained on both VF and clinical data (AUC, 0.89-0.93) showed better diagnostic ability than a model exclusively trained on VF results (AUC, 0.79-0.82; P < 0.001). CONCLUSIONS: A convolutional LSTM architecture can capture local and global trends in VFs over time. It is well suited to assessing glaucoma progression because of its ability to extract spatiotemporal features that other algorithms cannot. Supplementing VF results with clinical data improves the model's ability to assess glaucoma progression and better reflects the way clinicians manage data when managing glaucoma.