Overview

Coleridge, in collaboration with USDA’s Economic Research Service (ERS), worked on streamlining and updating the code used to process the Agricultural Resource Management Survey (ARMS) and U.S. Agricultural Productivity (USAgP) data products. Modernization efforts included a complete rewrite of the codebase used to produce these data products. This rewrite migrated the processing from proprietary to open-source tools (R), and implemented the best practices in software engineering, such as modular code, unit testing, and automated documentation. This effort aimed to help ERS achieve both increased reliability and reproducibility by using a more rigorous testing and debugging process in the workflow, enhancing continuity of operations by documenting the processes so that they can be rewritten or recreated, and increasing the usability of the data for end users.

Challenges & Objectives

The previous processing of the ARMS Phase 2 and Phase 3 and USAgP data products involved legacy code with multiple routines and several areas for improvement were identified. Over successive years of processing, idiosyncratic changes to the processing workflow had been accumulated. This led to a finalized code base that was increasingly complicated to understand and maintain. In order to provide for the future reliability of the data products, ERS was therefore looking to both streamline the statistical processing of the data and provide both researchers and maintainers with more comprehensive documentation describing the workflow.

The main objectives of the project were to 1) rewrite the data ingestion, preparation, and analysis codebase, 2) document and describe the process flow for data preparation and processing, 3) create a web-based codebook and navigation tools to help users understand the code used to produce each data product. The increased transparency these updates provided for the data processing resulted in a more maintainable and user-friendly codebase for each of the ARMS 2, ARMS 3, and USAgP data products. For code maintenance, future ERS developers will be able to more plainly understand the exact steps taken to source, ingest, and process the relevant data for each project. For end users, the web-based access to variable definitions and constructions will result in more well-informed research projects which implement appropriate decision rules more accurately aligned to those used in the USDA processing.

Process & Work

Coleridge rewrote the SAS modules used for processing the ARMS Phase III Cost and Returns Report data and for processing the ARMS Phase II Crop Production Practices data for populating the public-facing webtool. Coleridge also helped assess the existing code for the USAgP data product and helped create specifications for the rewrite using open-source tools (R). Coleridge also performed comparisons to ensure reproducibility. An Agile approach was used for creating specifications for each statistical processing module (what it needs to do), the business logic involved (why we need it a particular way), and the acceptance criteria (how we know it worked). The rewrite was organized in sprints of develop, test, demo.