Do users have confidence in the catalogues’ scholarship?

Most email survey participants expressed trust in the catalogues’ content, with no significant differences between the four catalogues reviewed. Many noted that the names of the four institutions instill trust that the text in the catalogues has scholarly value. Some participants may also have discovered that the catalogues (with the exception of the Johnson publication) are peer-reviewed, but they were not alerted to this fact, and focus group discussions suggested most users don’t find this information in a brief session. Despite this, 80% of survey participants said they would feel comfortable citing the catalogue in their work.

Participants who said they would not feel comfortable citing the catalogues gave a number of reasons, the most common of which was the difficulties in citing these resources. These participants said they had trouble finding the citation tools, couldn’t find page numbers, or just generally had difficulty or concerns about citing online publications correctly.

A smaller subset of participants had concerns about the quality of the text or its scholarly value:

I’m not sure. It doesn’t feel “scholarly” enough.

I spent my three minutes reading one catalogue entry. Not only were there some writing (grammar) errors, in my opinion it had insufficient citations, which makes me reluctant to use it for my own work. I would definitely use it to lead me to other sources, however, so it’s a good research tool.

I would assign it to my students as a class resource, but believe its content is too general info/encyclopedia-like to include in a scholarly article.

A few participants didn’t specifically question the scholarly value of the catalogues, but instead stated they had a general preference for print publications over digital. The remaining handful of respondents voiced concerns about the permanence of the text and links to the catalogues, general frustrations with the catalogue platform, and concerns about peer review.

Given time to explore the catalogues, however, many participants’ opinions of them changed. Seventy-two percent of respondents described the catalogues as “scholarly/academic resources” after a scan of the homepage, but this number increased to 81% at the end of the survey when participants were asked to describe the resource once more.

Focus group participants spent more time critiquing the scholarship of the catalogues and were sometimes divided in their opinions of the various works. Typos and grammar problems were called out as obvious issues. A focus group comprised largely of graduate students also expressed some general mistrust about the scholarly value of museum catalogues and the biases that the writing might contain:

That’s probably one of the most common places you run into really bad art history is in a catalogue, and if you are offering it as a resource to a student, it’s like a 50/50 shot that they’re going to get some useful or, and then it’s going to derail them.

I found a lot of the language to be off-putting and sort of old school [in the NGA catalogue]. Like the catalogue entry essay that we read about Rembrandt begins off the bat with a discussion of his genius, and the object entry that we read focused a lot on attribution. And I thought that the focus on the individual painter themselves and what a special hand they have behind the object read to me of this old school, genius-centered approach to art history that’s more celebratory than critical of the ways that the stories have been told in the past.

In some ways it’s like museum wall texts [exhibit labels], and so you don’t know if it’s being quirky or not—you know, if they’ve been told, “Don’t be too controversial,” or “Don’t give your opinion.” On the other hand, it may be completely out to lunch.

There are a number of ways, however, that the catalogues can instill more trust in their readers:

Digital catalogues should highlight authorship, peer review, and works cited

Not all focus group participants paid attention to the authors of the catalogues. When individuals did notice the authors, however, this increased their level of trust in the scholarship:

The other one I feel really confident citing is the Philadelphia one because it lists the author of everything that’s written on every page with a hyperlink to who that author is.

Arthur Wheelock, the primary author of the NGA’s Dutch Paintings catalogue, was called out multiple times in the focus groups as a trusted name in the field.

Most participants did not note whether or not the catalogues were peer-reviewed during their homework exercises. One group commented that they don’t generally think about peer review when using a website. One participant noted that the importance of peer review depended on the content she was looking at within the catalogues:

I guess if they were publishing analytical data I would expect things like that, but none of the technical sections were quite that in depth. It was mostly imaging and I wouldn’t necessarily really require that to be reviewed. I would base my assessment more on what references they had cited.

As the quote above notes, seeing a proper amount of citations to support arguments is another way users evaluate the scholarship of the digital catalogues, especially if the author is not well-known and respected:

I didn’t feel there was enough citation to support the arguments they were making…What we read from the NGA was written mostly by Arthur Wheelock. You can’t argue with Arthur Wheelock. He is really the top of the top scholarly [sic], and the PMA didn’t have top of the top scholars, but they were making arguments that to my mind were not sufficiently proven and therefore would not have been successful in peer review literature.

Address the temporality of digital resources through version transparency and DOI

The email survey demonstrated that only a few participants had strong concerns about citing the catalogues because of updates in the online environment. The rest have accepted that modern research relies on digital resources that may change over time:

I don’t know how that can be an issue in 2019 because what do we do but cite stuff on the web.

All the same, users were very interested in knowing how to access earlier versions of the catalogues and having an understanding of what had changed over time:

That’s what I really liked about the NGA’s, um, website—that they had the archived earlier versions and that you could track the history of that publication, that it was still all available, but you knew it was also very up to date.

A few participants also called attention to the DOI numbers featured in the PMA catalogue. These individuals—including a librarian and a museum professional with experience producing digital catalogues—said that the DOI imprint gives them more faith that they will be able to refer back to this resource over time:

So that to me engenders some trust. Even though citing online sources is thankfully more ubiquitous, I think having that DOI imprint…it informs others who might be reading a text that you’ve cited that it is a trusted resource.

Many users, however, did not know what the DOI imprint means. Museums may need to find a way to educate users in order for it to have a reassuring effect.