Data Management Planning tools: still immature?

26 Apr

I’ve spent the last few months looking at the JISC data management planning projects. It’s been very interesting. Data management planning for research is still comparatively immature, and so are the tools that are available to support it. The research community needs more and better tools at a number of levels. Here are my thoughts… what do you think?

At group or institution level, we need better “maturity assessment” tools. This refers to tools like:

  • DCC CARDIO for assessing institutional readiness,
  • the DCC Digital Asset Framework for understanding the landscape of data resources,
  • repository risk assessment and quality assessment tools like DRAMBORA, Data Seal of Approval, etc
  • security assessment tools including audits based on ISO 27000.

Some of the existing tools seem rather ad hoc, as if they had emerged and developed from somewhat casual beginnings (perhaps not well put; maybe from beginnings unrelated to the scale of tasks now facing researchers and institutions). It is perhaps now time for a tool assessment process involving some of the stake-holders to help map the landscape of potential tools, and use this to plot development (or replacement) of existing tools.

For example CARDIO and DAF, I’m told, are really tools aimed at people acting in the role of consultants, helping to support a group or institutional assessment process. Perhaps if they could be adjusted to be more self-assessment-oriented, it might be helpful. The DAF resource really needs to be brought up to date and made internally consistent in its terminology.

Perhaps the greatest lack here is a group-oriented research data risk-assessment tool. This could be as simple as a guide-book and a set of spreadsheets. But going through a risk assessment process is a great way to start focusing on the real problems, the issues that could really hurt your data and potentially kill your research, or those that could really help your research and your group’s reputation.

We also need better DMP-writing tools, ie better versions of DMPonline or DMP Tool. The DCC recognises that DMPonline needs enhancement, and has written in outline about what they want to do, all of which sounds admirable. My only slight concern is that the current approach with templates for funders, disciplines and institutions in order to reflect all the different nuances, requirements and advice sounds like a combinatorial explosion (I may have misunderstood this). It is possible that the DMP Tool approach might reduce this combinatorial explosion, or at least parcel elements of it out to the institutions, making it more manageable.

The other key thing about these tools is that they need better support. This means more resources for development and maintenance. That might mean more money, or it might mean building a better Open Source partnership arrangement. DMPonline does get some codebase contributions already, but the impression is that the DMP Tool partnership model has greater potential to be sustainable in the absence of external funding, which must eventually be the situation for these tools.

It is worth emphasising that this is nevertheless a pretty powerful set of tools, and potentially very valuable to researchers planning their projects and institutions, departments etc trying to establish the necessary infrastructure.

About these ads

14 Responses to “Data Management Planning tools: still immature?”

  1. Sarah Jones (@sjDCC) 29 April, 2013 at 10:52 #

    Hi Chris,

    Great post.

    I agree with you that some tools have emerged from beginnings unrelated to the scale of tasks to be faced. DAF was developed on a very small scale, 6 month project but has gone on to be used in lots of different ways. We’ve struggled to reflect this accurately. The content really needs to be updated to show how the tool and is actually being used in practice to help others learn from the improved approaches.

    There’s definitely a lot DCC can learn from the DMPTool work too, not least their partnership model. Martin works closely with the US team so we’ve got things in place to draw their lessons back into our work.

    Btw – DAF is the ‘Data’ (not ‘Digital’) Asset Framework :-)

    Sarah

  2. Chris Eaker 29 April, 2013 at 12:32 #

    I have thought that the DMPTool here does not provide much functionality over what one can do with a simple word processor. The only really useful thing it provides is the questions and prompts to get the user to think about data management. But these kinds of things are readily available on the web. The user still has to type in his or her answers. I know the DMPTool is going to go through more development and add some additional functionality, but as of right now, it’s not that useful.

    • Chris Rusbridge 29 April, 2013 at 12:36 #

      My feeling was that DMP Tool might manage to incorporate local guidance and reference to local capabilities better (or maybe more easily) than DMPonline’s “templte” approach. But I don’t know how to test this idea…

  3. mccutchv 29 April, 2013 at 13:17 #

    I think you are right to say access to better tools is necessary. We are all trying to set up environments to support researchers and ideally want to be able to offer proactive rather than reactive support.

  4. Jonathan Tedds 29 April, 2013 at 14:19 #

    I’ve been interested in using a risk analysis spreadsheet type approach with researchers/projects, as being developed for the DMvitals tool by Andrew Sallans, Sherry Lake and Susan Borda in the US. It won best poster at IDCC2012. See http://www.iassistdata.org/downloads/2012/2012_a1_sallens_etal.pdf

    • Chris Rusbridge 29 April, 2013 at 14:30 #

      I did get in touch with Andrew, and he told me (7 Feb this year) “DMVitals has been flying rather low, under the radar, as it’s more about internal operations than something for our researchers to see as an obvious, consumable service for their taking. Of course, there are elements which they will directly benefit from, but it’s not a self-serve model like DMPTool. So, anyhow, we are still indeed using and working on further development, and Sherry Lake and I even just finished a chapter about it this week.”

    • Jonathan Tedds 29 April, 2013 at 14:42 #

      Given limited institutional resources available to support data management planning, I think having something like this might be invaluable e.g. to assist institutional managers in prioritising intervention with researchers. But there will always be a need for researchers themselves, who best know their own datasets, to take responsibility and they will need tools they can use themselves to help with this.

  5. James Cheney 30 April, 2013 at 00:23 #

    Two thoughts:

    1. having played with DMPOnline very briefly, another thing that seems missing (slightly orthogonal to the group issue) is the ability to link the data management planning that you are supposed to do when applying for funding (in many areas) with what you actually do during the project.

    2. this reminds me of one of my first publications, on formal modeling of digital / informational preservation problems. Basically the idea was to model the problem of preserving some information as a game played between you and an unpredictable (but not necessarily malicious) environment, where your moves could be things like developing a new emulator or migration tool, and the environment’s moves could be things like new versions of Microsoft Word coming out and old ones disappearing. So given some preservation goals (quantitatively, a utility measure on information and available budget) + a lot of background information (including costs of different moves) one could try to maximize the value of information preserved over time. The idea of “self-assessing” a data management plan reminds me of this – a tool that could do this might follow a similar strategy.

  6. Luis Martinez-Uribe 30 April, 2013 at 06:33 #

    At the Australian National Data Service we have been playing around with instances of DMPOnline in the NeCTAR research cloud. Our aim was to demonstrate its features and potential value to interested universities. So far, we have had rather positive feedback.

    Both of your comments about the explosion of templates as well as the support model for DMPOnline change a lot when moving from a DMPOnline national service to an institutional one. When customizing DMPOnline for an institution, they can control the variety of templates through the admin area selecting what they want their researchers to see. They are also able to provide their own institutional guidance and point to the right local resources, a feature that is really well received. In terms of the support, maybe if there were more institutions deploying them locally then you may get a stronger community behind it.

    At the moment in Australia, research councils are not requiring data management plans as part of the funding application. This makes the benefit of DMPOnline a little less clear for Universities in Australia. Having said that, several institutions led by Flinders University are developing data management planning capabilities as part of the Redbox metadata store system (http://www.redboxresearchdata.com.au). Worth keeping an eye to.

    • Joy Davidson 1 May, 2013 at 14:14 #

      CARDIO was designed as a self-assessment tool to be used by institutional staff rather than external consultants. The key objective of CARDIO is to a shared understanding of perceptions of institutional maturity from a range of stakeholders rather than simply getting one person’s view. In this respect, the tool enables ratings on various elements relating to institutional infrastructure to be captured from a number of individuals and compared to help build consensus on areas where improvement is most needed. DCC are currently developing a version of CARDIO that will help HEIs to assess their progress towards EPSRC compliance. This version should be ready for use by June. I’d also recommend including UKOLN’s Community Capacity Framework Model (CCMF) in your list of maturity modelling tools and resources.

  7. Sam 3 May, 2013 at 02:27 #

    Hi Chris – this is an interesting topic and I have glad that you have raised it. I’ve been an observer in this space for a while and have some pretty strong views, which I will say up-front are my own and do not necessarily refelect the views of former, present and probably future employers!

    Some of the things that concern me (in no particular order) are:

    - the focus on projects – this makes sense in a funding agency context but in an institutional context I would argue that a group-oriented approach like the Data Curation Profiles from Purdue are more likely to result in lasting cultural change

    – the prevalence of the big white box for free text rather than offering a selection of options or structured data fields that would make it both less time-consuming for the person filling in and more useful (e.g. by enabling automated provisioning of storage space)

    - the assumption that the person creating the DMP knows (or can easily find out) the answers to the questions being asked and that the problem is simply that they are not written down (rather than the problem being that some of these areas are not well understood and/or ambiguous – I am thinking about ownership issues especially here)

    - lack of direct links to policies, services and contacts – this is an area that the DMPOnline has done very well in, with its customisable help

    - the duplication of information that is gathered through other business processes (e.g. grant application, ethics approval) and general lack of integration with core systems for identity management and research management – this is not a criticism of the tools themselves, except insofar as you would hope their roadmaps would include more work on APIs and import/export

    It’s a bit off topic for a discussion about the systems, but the other thing that has bothered me for a long time is our evidence base (or lack of) for DMPS in general. It’s possible that I am missing something, but I don’t see anyone asking (or attempting to answer) the question of whether DMPs are actually effective in changing behaviour, and even if they are, if the current focus on completion of a document is more or less effective than other kinds of activities (e.g. a facilitated data planning workshop). I would love to see this evidence, if anyone has it – it would make me feel a lot better about DMPs as something to include in an organisational strategy. As Luis mentions, the Australian funding agencies do not require a data plan. Yet, there is a widespread mis-perception that they do, and even those that do know they are not required have been optimistically forecasting their appearance in the very near future for at least the last 5 years. A number of Australian organisations have been introducing it as a requirement anyway, and there seems to me to be a powerful bandwagon effect going on whereby DMPs are being seen as a universally worthwhile thing to do, without a serious consideration about different jurisdictions’ broader policy and funding environments. A recent study by some researchers at QUT (published in Nature) showed that Australian researchers probably spent more than 500 person years on applications to our biggest funding scheme in 2012 (just 20% of which would be successful). If we did a cost- benefit analysis of time spent on DMPs, I wonder what we would find out?

    In my more cynical moments, I wonder if DMPs have taken off because they provide a quantifiable measure – number of data management plans completed – while other, possibly more appropriate measures of success (to what extent does the group have a shared understanding of their data management requirements and goals? how are data management practices changing for the better over time?) can’t be so easily quantified and require a lot more than deployment of a system.

    That’s probably enough career-limiting comment for today – thanks again for raising the topic!

  8. Dillon Clarke 10 May, 2013 at 07:09 #

    DMP tools like DMPonline have the potential to be used by institutions as a tool in bridging the gap between researchers and data services. They could either be hosted by the institution, or hosted by DMPonline and customized to incorporate the institution’s administrative requirements as well as those of particular funders. The contact details for the person to contact on different aspects of the data management planning could then be provided as part of the relevant guidance notes associated with specific questions. Additional questions around capacity and timings could be incorporated, allowing IT departments to better manage their resources.

Trackbacks/Pingbacks

  1. Data Management Planning tools: still immature?... - 22 May, 2013

    […] I've spent the last few months looking at the JISC data management planning projects. It's been very interesting. Data management planning for research is still comparatively immature, and so are t…  […]

  2. Data Management Planning tools: still immature?... - 20 September, 2013

    […] I've spent the last few months looking at the JISC data management planning projects. It's been very interesting. Data management planning for research is still comparatively immature, and so are the tools that are available to …  […]

Comments always welcome, will be treated as CC-BY

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: