Engaging stakeholders to refine models of state-level food self-reliance

Self-reliance measures the capacity of a geographic area to produce the food needed by its population. While the importance of food self-reliance, at even the national scale, is debated, the concept remains useful for evaluating the capacity of local and regional food systems to meet current and future human food needs. Modeling can estimate the capacity of geographic areas to supply food to their own population, but such approaches may be mere academic exercises if not perceived as credible and useful to stakeholders. This paper reports on an effort to engage stakeholders in refining a model. a * Corresponding author: Christian J. Peters, Ph.D., Friedman School of Nutrition Science and Policy, Tufts University; 150 Harrison Avenue; Boston, MA 02111 USA; +1-617-636-6908; Christian.Peters@tufts.edu b Jennifer L. Wilkins, Ph.D., R.D., Division of Nutritional Sciences, Cornell University, 305 Martha Van Rensselaer Hall, Ithaca, NY 14853 USA. Dr. Wilkins is now at the Department of Public Health, Food Studies, and Nutrition; 417 Sims Hall; Syracuse University; Syracuse, NY 13244 USA; jlwilk01@syr.edu c Scott R. Rosas, Ph.D., Senior Consultant, Concept Systems, Inc.; 136 East State Street; Ithaca, NY 14850 USA; srosas@conceptsystems.com d Brenda K. Pepe, Client Services Consultant, Concept Systems, Inc.; 136 East State Street; Ithaca, NY 14850 USA. Ms. Pepe is now at eCornell, Cornell University; 950 Danby Road, Suite 150; Ithaca, NY 14850 USA; bkp37@cornell.edu e Jamie Picardy, Ph.D., Friedman School of Nutrition Science and Policy, Tufts University; 150 Harrison Avenue; Boston, MA 02111 USA. Dr. Picardy is now at Geography and Regional Planning, Mount Ida College; 777 Dedham Street; Newton, MA 02459 USA; jpicardy@mountida.edu f Gary W. Fick, Ph.D., Section of Soil and Crop Sciences (Emeritus), Cornell University; 507 Bradfield Hall; Ithaca, NY 14853 USA. Disclosure This research was supported in part by funding from the W.K. Kellogg Foundation, grant number P3008987. Journal of Agriculture, Food Systems, and Community Development ISSN: 2152-0801 online www.AgDevJournal.com 56 Volume 6, Issue 4 / Summer 2016 Small groups of stakeholders were gathered in oneday workshops in four states to learn and provide feedback about two ways of modeling food systems: a simulation model of dietary land requirements and human carrying capacity (foodprint), and a spatial-optimization model of the potential for population centers to meet food needs locally (foodshed). Workshop participants engaged in smalland large-group discussions to critically assess the value of the models for food system planning and policy. Formal evaluation gauged the utility of the workshops as learning environments and the participants’ opinions of the models as food system planning tools. Results indicate that the workshops successfully taught participants about the models and elicited feedback on the relevance of the models to food systems planning. However, assuring relevance and application of food system models in local and state planning will require a deeper level of engagement and a greater time commitment from both researchers and stakeholders than a one-day workshop can accomplish.


Introduction
A consensus is emerging that meeting global food needs sustainably will require changes to both agricultural production and food consumption (Foley et al., 2011;Godfray et al., 2010;Hoekstra & Wiedmann, 2014).Such complex issues arguably require a "food systems" approach that includes processing, distribution, retailing, and consumption of food in addition to agriculture (Ericksen, Ingram, & Liverman, 2009).Furthermore, the National Research Council (2010) recently concluded that continued progress in agricultural sustainability will require both incremental and transformative strategies.In this context, local and regional production systems represent critical opportunities to address both food security and sustainability.Local and regional food systems generally meet the definition of "transformative" strategies since they are significantly different from the predominant food system.In addition, local food systems emerged, in part, in response to long-term concerns about the viability of farms and rural communities, the energy use and emissions associated with long supply chains, and an interest in fresh, nutritious food (Martinez et al., 2010).Regional food systems, with a wider geographic scope, may bring to scale some of the benefits of shorter supply chains (Clancy & Ruhf, 2010).While the benefits of local foods remain a matter of debate, it is important to recognize that systems in place today likely have not leveraged all opportunities for efficiency (Schönhart, Penker, & Schmid, 2009).The potential of local and regional food within the U.S. food system to meet food needs remains an open question.
Models provide a means of sorting through this complexity.As Canham, Cole, and Lauenroth (2003) describe, quantitative models have three distinct purposes in science: observation and experimentation, synthesis and integration, and prediction and forecasting.A variety of approaches have been used to study self-reliance in food systems at multiple spatial scales.Net balance studies have assessed self-reliance by calculating the ratio of historical food production to food consumption based on available secondary data (Cowell & Parkinson, 2003;Griffin, Conrad, Peters, Ridberg, & Tyler, 2014;Herrin & Gussow, 1989).Scenario modeling has been used at the state scale to estimate the number of people who potentially could be fed from local land resources under different diets (Peters, Wilkins, & Fick, 2007) or different assumptions regarding the quality of land suitable for production and area reserved for energy production (Kim, Burnett, & Ghimire, 2015).Foodshed mapping uses spatial estimates of the productivity of agricultural land and a variety of optimization algorithms to determine the distance in which population centers theoretically could meet their food needs at the state, regional, and national levels (Hu, Wang, Arendt, & Boeckenstedt, 2011;Peters, Bills, Lembo, Wilkins, & Fick, 2009;Zumkehr & Campbell, 2015).While the methods differ, these examples all share a common purpose: to integrate knowledge about food needs and production capacity in a way that yields insight about potential self-reliance on local and regional food.
In modeling, knowledge does not automatically lead to action.Land use models, for example, use sophisticated techniques to attempt to capture the processes behind land use change, but they are difficult for stakeholders to understand and hence often fail to influence decisions (Sohl & Claggett, 2013).Experts in sustainability science and integrated assessment, analytical approaches that use modeling, argue that including stakeholders in the research process may improve the modeling quality and applicability to real-world problems (Mauser et al., 2013;Miller et al., 2014;Salter, Robinson, & Wiek, 2010).However, attaining meaningful interaction between researchers and practitioners is difficult, since few transdisciplinary studies achieve a high level of engagement with practitioners (Brandt et al., 2013).
Recognizing this conundrum, the Foodprints and Foodsheds: Tools for Evaluating the Sustainability of Dietary Patterns and the Geography of the Food System project was designed to engage stakeholders in a process of adapting existing food system modeling approaches to new geographic areas and different spatial scales.The project used two modeling frameworks developed to study New York state food systems to establish a standardized process for studying three additional states and the conterminous U.S. The first approach, the "foodprint" model, estimates the land area required to meet a person's annual food requirements and the capacity for a geographic area (such as a state) to feed its population from available agricultural land (Peters et al., 2007).The second approach, the "foodshed" model, uses a combination of geographic information systems and optimization techniques to map potential, local foodsheds for individual population centers (Peters et al., 2009;Peters, Bills, Lembo, Wilkins, & Fick, 2012).These approaches lie within a relatively new area of work, categorized as foodshed assessment by Freedgood, Pierce-Quiñonez, and Meter (2011).
The foodprint and foodshed models were developed to examine fundamental questions about the potential capacity of statewide land resources to support local and regional food systems.Like most models, the approaches were initially developed within an academic environment, open to input from researchers but relatively isolated from the input of practitioners.Having demonstrated that the models were technically feasible, we believed the next step was to test how well these approaches resonate with stakeholders.Through the Foodprints and Foodsheds Project, we sought to gather stakeholder feedback on the applicability of these tools to food system planning.
To this end, we engaged a group of stakeholders in a workshop approach in each of four target states (Michigan, Mississippi, New Mexico, and New York).While workshops are just one of a variety of methods used to engage stakeholders (for example, see Rowe & Frewer, 2005), they are a primary mechanism for gathering input from stakeholders in a related type of research called participatory integrated assessment (Salter et al., 2010).Further, workshops are an opportunity for participants to learn more about a topic of a scientific or technical nature, consider relevant evidence, and discuss this evidence with other participants from varied backgrounds before presenting their opinion (Ableson, Forest, Eyles, Smith, Martin, & Gauvin, 2003;Evans & Kotchetkova, 2009).
The purpose of this paper is to describe lessons learned on the value, process, and challenges of involving stakeholders in food system model development and research.To this end, we share the method by which we assessed the success of our workshops and the results of the evaluation.While some of the findings are specific to the Foodprints and Foodsheds Project, we reflect on the experience to draw out lessons of general value to others interested in engaging stakeholders in modeling research.

Methods
Workshops engaging selected food systems stakeholders in four locales (East Lansing, Michigan; Jackson, Mississippi; Albuquerque, New Mexico; and Ithaca, New York) occurred over a three-year period (2009)(2010)(2011)(2012).The workshops were convened to gather stakeholder input on how the foodprint and foodshed models could be adapted and applied in each state.New York was selected as a workshop location because of the geographic focus of the original modeling research (Peters et al., 2007;Peters et al., 2009), and the other states were selected because they are sites of active programming by the project's funder.The workshops were planned and implemented by the research team of the Foodprints and Foodsheds Project, and formal evaluation was performed by an external evaluator to assess the engagement of stakeholders in each workshop.Workshop design and implementation and the evaluation approach are discussed in detail below.

Workshop Design and Implementation
Four full-day workshops were held over the threeyear period, one at each of four different locations.This timeline was chosen to give the research team time to replicate the foodprint and foodshed modeling approaches for the target state while simultaneously planning the workshop.Each workshop was preceded by an extensive preparation phase in which the research team developed presentation materials, assembled a list of potential invitees, chose a venue, sent invitations, and shared pre-workshop materials with prospective participants.Several evaluation processes followed each workshop, and the research and evaluation teams met during the months after each to review results and plan adjustments to subsequent workshops.Spacing workshops by approximately one year allowed for this iterative process.
The workshops were designed to facilitate interactive dialogue by gathering small groups of selected participants who represent the communities of interest (Rowe & Frewer, 2005).We identified individual stakeholders who were (a) interested in learning about the food system models, (b) able to judge the value of the models for informing food, nutrition, agriculture, and food system policy and planning, and (c) likely to employ them in their locales.Research team members relied on their pre-existing professional networks to develop the invitee list for New York.For the other states, the team worked with local partners to assemble lists of invitees.Attendance at the workshops ranged from 8 to 21 individuals, and collectively the workshops involved 66 participants.The Michigan and New Mexico workshops had the most attendees (n = 21 in each), followed by New York (n = 16) and Mississippi (n = 8).
The purpose of each workshop was to elicit stakeholder input for further refinement of both food system analysis models: the foodprint model and the foodshed model.The approach at each workshop involved several interactive activities to increase stakeholder understanding of the intended applications of the models in "real world" food systems analysis and planning, and for the researchers to gain an understanding of the strengths and weaknesses of the models in addressing food system analysis and planning needs as perceived by the stakeholders.The stated objectives of the workshop were to (a) share information about the foodprint and foodshed models; (b) provide an opportunity to consider potential applications of the models; (c) identify opportunities and barriers to enhancing application of the models; and, (d) elicit feedback on aspects of the models.
Each of the workshops, entitled, "Developing Tools for Food System Analysis and Planning," followed a similar agenda.At the start of the workshop, one of the project team members welcomed participants, provided an overview of workshop goals and issues to be discussed, and outlined the team's motivations for pursuing the modeling work.Following the overview, a presentation was given on the development and current use of the foodprint and foodshed models.A facilitated group discussion then allowed participants to provide initial impressions and feedback on what was presented and to ask questions.After lunch, small group discussions about how the food system assessment and planning tools could be put into action were followed by reporting back to the larger group.The New Mexico and Mississippi workshops also included time for a hands-on exercise to explore the foodprint model, 30 minutes for small-group discussion, and 30 minutes of reporting back to the larger group.The workshop approach was replicated for each of the four locales, with the project team making the necessary adjustments to fit the context.

Evaluation Approach and Tools
Evaluation was used to assess how well stakeholders were engaged and to determine if and how the workshop objectives were met.A mixed-methods evaluation approach was designed, where qualitative and quantitative data were collected and analyses integrated (Figure 1).We viewed the evaluation as opportunity to learn about the utility of our approach and anticipated that it would provide timely feedback at critical points during the project to guide workshop improvements or adjustments.The evaluation approach included four methods: workshop survey, observations, follow-up survey, and interviews.These methods and tools are described briefly below, and technical details are provided in the Appendix.

Workshop Survey
A brief survey was distributed to participants and collected at the end of each workshop.The survey gathered information on (a) the participants' perceived level of knowledge of the foodprint and foodshed models; (b) their opinions on how the workshop was run; (c) the amount of time they spent reviewing the materials sent prior to the workshop; (d) their intention to connect with new colleagues once the workshop was over; (e) their professional food system role; and (f) their willingness to be contacted for a follow-up interview several weeks after the workshop.Where appropriate, participants were asked to consider their understanding of the models at two points in time: before they came to the workshop and now that they had participated in the workshop.

Observations
A member of the evaluation team observed the behaviors and actions of participants during the workshops.Audience reactions and responses to general presentations were observed and manually recorded by a member of the evaluation team as the presentations were delivered.During the breakout sessions, the observer rotated through the small groups to capture participant reactions and responses.

Follow-up Survey
Six to eight weeks after each workshop, a short follow-up survey was conducted.The survey contained a combination of fixed response and openended questions to assess the degree to which the workshop influenced participants' work and their conversations about food systems while the experience of the workshop was still fresh in their minds.

Interviews
Post-workshop interviews were conducted with a subset of participants who indicated on the follow-

Workshop Surveys
Across the four workshops, survey responses were received from 59 of 66 (89%) participants.Selfascribed roles in food systems work (Table 1) indicate that the participant selection process succeeded in gathering professionally diverse groups of people at the workshops.While the composition of the audience varied from location a "Agree" includes a response of either "Tend to agree" or "Agree" on the scale.See Appendix.Retrospectively, relatively few participants indicated they had knowledge of the workings, application, and data used in the footprints and foodsheds models before the workshop (Table 2).Indeed, no more than 12% of the participants agreed that they possessed the aforementioned knowledge prior to being exposed to the information in the workshop activities.A significant difference in the number of the participants indicating retrospectively they had knowledge related to the models at the end of the workshop was found across all 8 items (see Table 2).Between one-third and onehalf of participants agreed that compared to what they knew before, they had specific knowledge related to the models at the conclusion of the workshops.
Over three-quarters of the participants agreed that the logistics for the workshop were well executed and the sessions were well facilitated (Table 3).However, a significantly smaller number agreed that enough information was provided during the workshop and in the pre-workshop package for participants to be able to answer all of the questions posed in the breakout group sessions (χ 2 [1] = 4.77, p < .05).Nevertheless, greater than two-thirds of the participants agreed that each of the four objectives of the workshop were met (Table 4).The pre-workshop communications gave me the information I needed to learn about and prepare for the workshop.22 (37.3%) The workshop program engaged me in active learning related to its goals.37 (62.7%) The workshop sessions were well facilitated.43 (72.9%)The logistics for the workshop were well executed.
46 (78.0%)The workshop provided me with enough information to answer all of the questions in our workgroup.22 (37.3%) The materials provided to me during the workshop were useful.34 (57.6%)As a result of this workshop, I am likely to use the information in my professional role within the food system.24 (40.7%) a Agree includes a response of either "Tend to agree" or "Agree" on the scale.See Appendix.To share information about the Foodprint and Foodshed Models 44 (74.6%)To provide an opportunity to consider potential applications of the models 44 (74.6%) To identify opportunities and barriers to enhancing application of the models 39 (66.1%) To elicit feedback on aspects of the models 45 (76.3%) a Agree includes a response of either "Tend to agree" or "Agree" on the scale.See Appendix.Mapping potential foodsheds in New York State: A spatial model for evaluating the capacity to localize food production.17.52 Foodshed analysis and its relevance to sustainability.13.12 Testing a complete diet for estimating the land resource requirements of food consumption and agricultural carrying capacity: The New York example.

13.34
Foodprints and Foodshed Project: Tools for Evaluating the Sustainability of Dietary Patterns and the Geography of the Food System-Project Summary 10.88 a Agree includes a response of either "Tend to agree" or "Agree" on the scale.See Appendix.
Participants, on average, spent nearly an hour reviewing pre-workshop materials (Table 5).Average time spent on each individual document was greater than 10 minutes and less than 20 minutes.On average, participants spent more time reviewing each of the three journal articles than in reviewing the one-page project summary.More time was spent viewing the longer documents, but the relationship was not directly proportional.

Workshop Observations
Participants were actively engaged and attentive during the presentations.A substantial amount of dialogue and discussion about the models occurred among participants as well as with the researchers at the workshops.Breakout groups were often observed to be highly productive, with several group members discussing the question posed to the group.When participants had an opportunity to explore the foodprint model, there were many "aha" moments and surprised expressions when participants adjusted the calculations for dietary proportions of different foods in the protein food group.There were, however, several instances when group members were unclear about the directions and questions assigned to the breakout group, resulting in puzzled looks and frustrated exchanges about the lack of clarity.However, in these instances the research team member quickly remedied the situation and reengaged the group.In general, the participants expressed real interest in the research and activities that were shared at the workshops.
While the observations uncovered positive reaction and responses from workshop participants, there were instances of negative reactions as well.For example, during the New Mexico workshop there was some resistance and discomfort expressed by the participants.While learning about the foodprint and foodshed modeling approaches, some participants vocally disagreed with the premise that land, in particular Native lands, may be repurposed to meet population food demands due to their concerns about food and land sovereignty.Facial expressions and body language observed by the evaluation team also indicated that there was a "disconnect" between the perspective and assumptions that framed the research team's work and the contextual and cultural issues considered by workshop participants.The presentation was paused to allow for an extended conversation between participants and the researchers about the assumptions inherent in the approaches.Ultimately, the positive and negative feedback observed during the workshops provided fruitful material to consider in further development and dissemination of the model.

Follow-up Surveys
The post-workshop, online survey was completed by 35 of 59 (59.3%) workshop participants.Results indicated that overall the workshops were supportive to participants in several areas (Table 6).Based on the number of participants rating the workshops as "a great deal," the two highest ranked items were (a) increasing the awareness of emerging research, and (b) encouraging the sharing of information with local groups.The two lowest ranked items were in response to the workshop's role in (a) increasing professional networks, and (b) exposing participants to language for explaining the model to others.Nearly half of respondents (45.7%) agreed that materials were useful for (a) providing evidence for food system planning and change, and (b) providing clarity in explaining key concepts about foodshed and foodprint models (Table 7).However, fewer respondents agreed that the materials were useful for the more practical activities (e.g., your own work; research questions and projects), and the lowest ranked item was in response to the materials being useful in program planning and development.
Responses to the open-ended questions confirmed the positive ratings, revealing that participants valued the new knowledge gained and the opportunity to network with potential colleagues.Furthermore, participants suggested that the research team might consider building in more time for reflection on the relevance the new information to one's work, additional time to share experiences with one another, and the provision of more summary materials, such as a final document of ideas presented and copies of the presentations.More than half (54.3%) of respondents indicated they had follow-up conversations about the workshop materials with other workshop participants.Of those who said they did not have follow-up conversations, nearly half (46.7%) indicated that they planned to do so.

Interviews
Of the 17 workshop survey respondents who agreed to an interview, 15 were interviewed by a member of the evaluation team.These individuals self-selected to be available for a follow-up interview, and so their views may not be representative of the larger population of participants.Nonetheless, concurrent with the purpose of the interviews, the input gathered from those who volunteered represented a reasonable appraisal and provides further insight on the strengths and weaknesses of the workshops.Several major themes emerged.
Novel approach to food systems analysis First, interviewees indicated that the models presented were a unique and innovative approach to food systems research.The workshops provided new terminology and visualizations to describe food systems analysis to participants.Interviewees appreciated the chance to network and connect with others in the field, with the small group breakouts contributing much to this collaborative experience.In hearing about the models, interviewees viewed the application of the models as promising and the prospects of its usage very exciting.Two quotations from the interview capture this enthusiasm: The most obvious advantage of using these models is to open people's imaginations to thinking in new ways.
I would love to have the tool in the classroom to help with teaching purposes.

Expanded view of sustainability
A second major theme to emerge was that the sustainability focus encouraged a broader view of the food system.Topics covered during the workshops encouraged thinking about food systems planning and the need for tools to aid in local and regional sustainability.Specifically, the workshops helped make a connection between food systems sustainability, dietary patterns, and decisions regarding meeting food demands with local and regional production systems.Interviewees reported feeling encouraged by the expression of a broad vision about this being a response to feeding the nation and not limited to addressing colloquial issues.As one interviewee put it, "It changed the way I thought about what impacts a food system's sustainability.…Inow look at this issue in a much broader framework."

Sophistication of models
A third major theme had to do with the complexity of the models.According to interviewees, the model is complicated and not easily or quickly explained.Based on interviewee feedback, it takes time to appreciate its nuances.They reported difficulty with identifying and obtaining local data needed to contextualize the models.This potential barrier may preclude data aggregation and replicating results meaningfully in their locales.For many of the interviewees, the models were considered too abstract and challenging to apply to food systems practice.Two quotations from the interviews encapsulate these concerns: The data is really hard to aggregate and get a handle on.The model is complicated and draws on his academic research rather than practitioner work, and it is a huge challenge without years of academic research behind you.
This data is hard to come by and I cannot really mimic the models.I do not know how to do this on my own because the method is very complex.

Gratitude for engagement
Fourth, several interviewees expressed appreciation for being asked to be a part of what they perceived to be an innovation in food systems analysis.As one interviewee explained, "For me, the highlights were numerous.The work was interesting and innovative.The conversation was engaging.Networking is always high on my list."The smallgroup settings allowed for sharing of perspectives in a way that brought the issues to life and facilitated learning.Interviewees acknowledged the value of the opportunity to network with new contacts and have rich conversations where different perspectives on the models and their applications were shared.In particular, interviewees noted that the contribution of multiple voices clarified assumptions and pushed perspectives of those involved in the workshops, including the research team.
Accessibility for general food system practitioners Finally, interviewees remarked that much work remains to make the materials adequate for public dissemination.For example, one interviewee claimed, "The workshop didn't respond to the practical issues that I would need to know to make better use of the information presented."Interviewees felt the materials were too technical for general food system practitioners and emphasized that materials and discussion points need to be tailored to specific sets of target audiences.The interviewees suggested that the team should consider how web technology can be used to promote the concepts and disseminate materials.Interviewees also indicated that they feel there may be a need for additional data, such as local sources of food, economic, and jobs data, land development information, climate change data, and other geospatial information that could increase the relevance of the models.
Several interviewees suggested that a forum to discuss the model might increase its application.Policy-makers, practitioners, and funders could talk about how the models pertain to their work and the decision they make regarding food systems.Making the models available for others to explore and manipulate was seen as a way to increase the visibility of the information and materials from the workshops.Examples and scenarios could be made more widely available to stakeholders to improve understanding and application of the materials across a broad range to food systems professionals.

Degree To Which Objectives Were Met
The workshops were successful as a mechanism for communication.Participant understanding of the modeling approaches clearly increased as a result of the day spent with the research team.In addition, participant ratings of the workshops indicate that the stated workshop objectives were met.Some participants indicated they would likely use foodprint and foodshed modeling research in their own professional work.Participants felt prepared and engaged in the workshop, which highlighted and supported the interactive design.Observations of interactions and behaviors during the day also confirmed that participants were interested, engaged, and attentive.
Follow-up information from workshop participants indicated they continued to support their initial assessment that the workshop was successful.Specifically, participants agreed strongly that the workshop increased knowledge; provided new language and awareness of emerging research; fostered networking, sharing, and new professional interactions; and supported continued dialogue regarding the information presented at the workshop.Furthermore the workshop participants found the materials to be most useful in sharing with others and providing clarity around the key concepts covered at the workshop.

Lessons Learned
In many ways, the workshops were executed in a manner consistent with recommendations from the literature.As Cohen et al. (1998) suggest, the models were run for specific geographic areas yet placed in the context of overarching societal issues.As Miller et al. (2014) advise, participants came from a range of backgrounds, including many outside academia.We aimed for a high degree of interaction, both among participants and between participants and the research team.These were achieved at each of the workshops.We also gathered input from workshop participants that has proved useful in further revision and refinement of the models.Given the limited participation of policy-makers at the workshops, the value this stakeholder group sees in how the models might inform policy could not be assessed.Had the participants come to the workshop with preformed food systems issues and questions of relevance to their state and had the models adjustments been completed for the appropriate state contexts, the workshops may have provided an opportunity to generate data of interest to policy-makers.Review of participatory approaches indicates that integrated assessment has not generally influenced policy processes (Salter et al., 2010), so this conclusion is not surprising.However, the team learned some important lessons on stakeholder engagement for future work: 1. Building strong participation takes time.A single, full-day workshop was sufficient for the purpose of improving understanding and gathering initial impressions of the modeling work.However, to get deeper insight on the applicability of the model for planning purposes, either a longer workshop or multiple sessions would be needed.
2. Participant composition is critical.A broad audience was appropriate for gathering reactions to the model as tools for envisioning the potential for local and regional food systems.However, more targeted groups would be necessary to involve participants in the actual adaptation of the model to a geographic area, such as meeting with agricultural scientists, extension agents, and farmers to discuss assumptions about crop and livestock productivity.
3. Setting expectations appropriately avoids disappointment.Foodprints and foodsheds models are useful tools for visioning.However, focusing solely on land, these tools cannot address issues like economic viability or access to water rights, for example.Participants should understand the limits of a model from the outset to avoid dashed expectations.
4. Inviting open-ended and critical feedback builds trust and improves dialog.Participants came to the workshops with deep experience from different vantage points in the food system, but only a few had experience with modeling and most did not know the researchers.Workshops should include opportunities to build rapport between the research team and the participants.
5. Use evaluation to ensure that expectations of stakeholders are met.The evaluation was planned with a consultative approach and conducted in ways to (a) make major decisions about the delivery of project materials, (b) contribute to improving relevance for stakeholders, and (c) generate knowledge on lessons learned for future application.Replicating the evaluation approach across sites enabled the team to focus on timely adjustments, as well as to aggregate information to describe the success of meeting goals and objectives.

Conclusions
Enhancing the sustainability of food systems requires both new methods of analysis and the translation of knowledge into action.Models provide a means of integrating data to better understand food systems.Yet unless they are comprehensible, credible, and relevant to stakeholders, they will remain solely of academic importance.Our experience using consultative workshops indicates that complex models can successfully be described and critiqued in a workshop setting.However, one-day, stand-alone workshops do not provide a chance to iteratively improve models to make them better reflect local conditions and provide information that is more directly applicable to the day-to-day work of practitioners.Interaction with stakeholders that leads to application of models in decision-making would require ongoing engagement over a longer term.
Future research with food systems models can be enhanced by integrating stakeholders into the research process, but considerable forethought must be given to the roles stakeholders will play and the time and resources needed to support such interaction.In practical terms, this likely means that key stakeholders and researchers must work together from proposal design through project completion in order to achieve this deeper level of engagement.It also means that sufficient common ground and adequate resources are needed to support collaboration between researchers and stakeholders over extended periods of time.

Figure 1 .
Figure 1.Work Flow and Relationships Between Workshop and Evaluation Activities Boxes indicate discrete activities in the workshop design and the implementation and evaluation approach.Arrows represent flow of information.Research team activities are shaded in gray.Evaluation team activities appear in white.Joint activities appear in striped boxes.
willingness to be interviewed.The interview protocol focused on five main questions and three follow-up questions concerning the workshop's impact in the following areas: (a) the possibilities and challenges of the presented research to inform food system policy or planning; (b) how the workshop was reaffirming and helpful to the way participants think about food system sustainability; (c) what participants felt about the utility of the models in their own food systems work; (d) opinions about public dissemination of workshop materials; and (e) suggested refinements to the models for practical user application.

Table 2 .
Retrospective Pre/post Workshop Survey Responses a p < .001

Table 1 .
Professional Roles in Food Systems Work of Workshop Participants a a Workshop participants could indicate more than one professional role.b "Other" responses include University Administrator; Developer/Evaluator; Administrator; Relocalization; Funder; Foundation Executive; University Outreach; Public Health; Artist; Freelance Educator; Farm-to-School; Community Foundation

Table 3 .
Workshop Survey Responses Regarding the Perceived Quality of the Workshop (N=59) a

Table 4 .
Number and Percent of Participants "Agreeing" Workshop Objectives Were Met (N=59) a

Table 5 .
Time Spent by Participants Reviewing Materials Prior to the Workshops

Table 6 .
On-line Survey Responses Regarding the Effectiveness of the Workshop (N=35)

Table 7 .
Online Survey Responses Regarding the Usefulness of Workshop Materials (N=35)