Research and Evaluation Task Force
February 21, 2020
PRESENT: Luke Shorty, Lisa Phelps, Nate Hinchey, Maryalice Crofton
- Consideration of 2 questions CNCS omitted from 2017 supplement questionnaire.
This discussion was very short. Maryalice did some investigating about why the questions were dropped. Researchers identified them as questions that were not likely to elicit reliable answers. This type of survey needs to focus on behavior rather than sentiments. If the Commission does support adding responses by administering the survey, it will not add these.
- Discussion of USM proposals for survey and report as well as NCOC responses to task force questions.
The data and survey units at University of Southern Maine collaborated on a proposal to administer the survey used by the federal government and available as public information. The goal is to add to the low federal sample size for Maine so that better analysis of all the civic health indicators could be done, including substate reports. The 2017 report only had 400 Maine respondents and at least 1,000 are needed.
In a follow-up to the last meeting, Jeff Coates from the National Commission on Citizenship (NCOC) shared that they get their data from the Corporation for National and Community Service and it is already analyzed. Because the survey administered in the Current Population Survey supplement was radically revised for the 2017 data, it is not possible to combine 2017 responses with past years. That is one cause for the inability to do substate reports. It will take several rounds of administering the survey (done every 2 years) to get enough responses to combine years. The survey was last done in fall 2019. Without adding to the data through a Maine supplementary survey, it could take 6 to 8 years unless the state sample size increases. NCOC confirmed it manages creation of a civic health report for $35,000.
Maryalice is reaching out to the analyst at CNCS to get some specific information about when the 2019 data will be available, what the Maine response size was in 2019, and whether the response rate (percent of people answering) changed from 2017.
University of Southern Maine’s data and survey units have described concerns about conducting the survey and adding the responses to the national data pool for Maine. These include the fact the added data would be collected six months later than the original; it is possible someone who answered the federal survey would also answer the state survey; it is not possible to interview residents under 18 years of age as the federal survey does; and the response rate for an in-state effort is more likely to be 7% rather than the much higher rate the Census experiences. The difference is due to their method of developing a set of people in the general population to survey. USM was concerned about the number of footnotes or cautions that would need to be part of any report in which conclusions were drawn from data that was a mix of federal and state collected. Task force members were not as concerned about having the cautions in a report, especially if 2019 data was being added to data collected by June 2020.
Attention turned to the USM cost proposals. The most expensive USM proposal was dismissed as not one the Commission could contemplate. Members discussed potential partners who might match the Commission’s funds so the most reasonable proposal could move forward. As a follow up to this meeting, a one-page description of the project will be finalized and shared with members who will reach out to potential partners.
- Setting regular meeting time
Going forward, the task force will meet the 4th Friday of each month at 1pm. The regular meeting venue will be skype.
- Next steps
-- After the one page description is available, members will reach out to potential partners.
-- Maryalice will follow up with both USM and the CNCS analyst.
-- It was agreed that regular updates would be provided between meetings so the project could move forward.
NEXT MEETING: March 27 at 1pm.