Greg Leppert keynote on data and knowledge institutions in AI: transcript

Key points

• AI rests on research, compute and data, and public interest institutions have the most access to data, not to compute.
• The Institutional Data Initiative released about a million public domain books scanned at Harvard Library and is now working with Boston Public Library on historic newspapers.
• New open source OCR models read newspaper ads far better than Tesseract, but cleaning text risks losing nuance, which he compares to autotune.
• He wants cultural institutions to be authoritative sources and experts in AI, and wants AI to depend on them, using data leverage.
• He says data leverage can seek consultation, accommodation and compensation, and that restrictions on data should not last long.
• He proposes a regulatory advantage for libraries: publishing public domain works with reasonable takedowns instead of statutory damages.

Chapters

1. 0:04Introduction and opening remarks
2. 4:57AI as research, compute and data
3. 7:06Institutional Data Initiative and public domain books
4. 8:52Newspaper OCR compared with Tesseract
5. 11:12Autotune and losing nuance in cultural data
6. 12:42Data leverage and the role of cultural institutions
7. 16:00Q&A: access restrictions on the books dataset
8. 18:25Q&A: the role of non-experts
9. 21:07Q&A: Wikimedia's leverage over AI companies
10. 23:22Q&A: corporate rights holders and library advantage

Full transcript

0:04So I think we're good to go. Did you enjoy your meal? Yes? Okay, great. So we're going to our first keynote with Greg LaPert. Data is everything why knowledge institutions are the future of AI. Greg is the Executive Director of the Institutional Data Initiative at Harvard Law School Library, a research initiative that helps libraries, universities and government agencies. share their collections as high-quality data sets.

0:37He's also the chief technologist at the Berkman Clean Center. And before that, Greg started a few tech companies in New York and Austin, played an in-art rock band, he's an artist also, and worked at a historic letterpress studio. And most important, he believes that most people truly want to make a positive difference. They're just looking for the right way to do it. please welcome greg leperts thank you so much uh and to be clear art rock is a term that you use when you make music that nobody understands including maybe even yourself so you just call it art uh okay so um From my perspective, I think we're in a golden age of information. And in fact, 25 years ago, I don't think any of this existed. To my knowledge, Wikimedia was not a thing. And now it is very much a thing. Now it's a cornerstone of how we access information across the world, how we synthesize it, how we come together to codify it.

1:47but some strange things are happening at the moment including that my clicker will not click there we go let's see if we can oh so now you're doing it now i'm tele-operating just like the the new uh i think maybe a new clicker is coming my way is that the Thank you.

2:52Yeah, this is actually part of the art rock piece. Back. So technical issues, and just a little second, I think we are good to go.

4:11You know, I once saw David Sedaris. Does anybody know David Sedaris? Yeah. And he lost his place in the middle of reading one of his passages. This was at the Reimann Auditorium in Nashville, and he paused for probably five minutes until he found it. And it was the most brilliant thing I ever saw, someone who just held a stage and did not give in to the desire to tell some anecdote from their life that might bridge the gap. I can go back and forth.

4:57So strange things are happening. One is that we have AI of some sort. Let's not quibble on what the term means, but we have something that we can talk to and get information from. And three years ago, if you had told me that we would have this, ChatGPT only launched, I think, in November of 2022. I remember, I'm an engineer, and if you're an engineer, friends will come to you and say, I have this great idea. I'm going to build a thing to do such and such. I had a friend come to me maybe five years ago. I'm going to build a platform to let me talk to it about botany. Can you help me with that? I said, good luck. We're not going to be chatting with AI anytime soon.

5:40and fast forward a couple of years, and here we are. It's very strange, but also strange is that we're defunding knowledge institutions at the same time. Does anybody know why we're defunding knowledge institutions? Anybody? I was hoping somebody would volunteer so you could tell me later. I have absolutely no idea, but when I think about AI, I think about it in these component parts. So just as like a big bucket of things, if you were to swirl it together, you've got research. that's kind of the underlying ideas that's the transformer and the attention mechanism that have made llm so powerful uh you have compute uh which you know as all the the computers just kind of munching things together and you have the data that that undergirds the whole thing that really goes into the top of the system now uh i work at harvard and i see all the time uh researchers leaving academia because uh there's so much interesting work to be done in industry and i see compute, I see, um, Companies regularly announcing commitments of hundreds of billions of dollars toward compute. And in public interest institutions, gosh, that's just kind of hard to come by. But data is something that we have a lot of access to. And that's no coincidence, really. But when I think about how AI works, I think about this cycle. Really, it's data feeds into AI, but AI is also feeding into data. We think about this from a slop perspective.

7:06You know, we're getting a lot of new information in the world. Some of it is AI generated and it is modifying what we know and how we know it in ways that we don't really anticipate. But I come from the position that I don't think this has to be a vicious cycle. I think this can be a virtuous cycle. And so this is what we're doing at the Institutional Data Initiative where I work. What we do is we partner with knowledge institutions, broadly defined, a lot of libraries, but we'd also be happy to work with museums, cultural groups, anybody. because we think data is a bedrock that can be a positive influence on the development of AI because it goes into that cycle and whether you're doing it from an open source perspective or you're doing it from a commercial perspective, there's a lot of incentive to have good data.

7:51So the first thing we released was a collection of about a million public domain books that were scanned at Harvard Library. We call that institutional books. You can download that today on Hugging Face. And we're just getting started. If everybody could turn their ringers on at exactly the same time, that'd be great. but now we're working with boston public library on a set of historic newspaper collections so this is a segmentation model that we trained because if anybody here has done ocr optical character recognition on newspapers they are difficult cultural objects there's a lot going on there and the more that you can decompose them a little bit perhaps the more that you can turn them into highly searchable collections now we've been trying to do this for a long time boston uh boston public library has digital commonwealth uh and there's a newspaper navigator in the us there initiatives across the globe to do this uh but they often break in a lot of ways uh when you're ocr-ing texts like this often it will just go right across the columns and so you'll end up

8:52with sentences slammed together uh and your search breaks and so very simply you know we've trained a model to segment it so that we can then begin to ocr at an individual level uh of a component so So here's an ad, and actually below this is, what is, you know, a few years ago was the state of the OCR. This is called Tesseract. A lot of people use it. You can see it's doing its best. rubber stamp. Rick Kinds. Instead of all kinds, Rick Kinds. Now, I used to be a letterpresser. I'm sure that when they did this, it was a great day in the studio. They just grabbed these great uppercase letters and slammed them together. But, you know, now we're struggling to get it done. But with new OCR models that have come out, and five open source OCR models came out just last week.

9:40We lived with Tesseract as kind of the de facto solution for the better part of a decade, and now five new models have come out. This is what we get instead. Rubber stamps, all kinds, printing of every kind. If you look, printing of every kind, it fades off there at the bottom. It's really hard for Tesseract to get that. Printing of every win. I think anything we print is a win, but Tesseract would agree. Here's another one, just to kind of show you how interesting things are getting. this is tesseract at the bottom an ad for storage uh there's a lot that goes wrong here and if you understand how the system works you can see how it's trying to compensate it's really working at a character level to try and recognize what the characters are but then when we look at uh you know this is dots.ocr you have um gosh something that's highly readable and highly searchable i think is one of the most important things here and so when we think about making better data it's not really about just making it better for ai it's about using ai to make it better for traditional patron systems as well.

10:41You plug this into the existing newspaper systems, search systems, and libraries, and suddenly this text is recognizable. Two musical instruments. The top is the mandolin from Chris Thiele. I don't know if you know Chris Thiele. He's a Grammy-nominated mandolin player for a group called Nickel Creek. He's fantastic. Look him up if you haven't listened to him. This is a beautiful mandolin made in 1924. It's a Gibson.

11:12at the bottom does anybody know what this is this is the first commercial autotune system also an interesting instrument i would say that has different effects uh within music now um one of the things chris says about his mandolin is that he's dropped it a number of times and there are new cracks in it and those cracks make for interesting intonations in fact the the instrument itself is evolving as he uses it and i would say that autotune also evolves music certainly from my perspective it can make things better some things better in certain ways but you have to be mindful about the way that you use it and one of the things i want to point out here is in that last ad that i showed you this is how they type set application at now if we go back a few slides we'll see gosh it did a great job dots.ocr uh it pulled everything out of here perfectly but it changed something. There's now a space there.

12:08And what happens to our culture if we start to auto-tune everything? We start to lose some of the nuances. Do we lose the ability to create compound words? Do we lose the ability for an accident? I assume this is an accident. I got pretty lazy when I was a typesetter. Do we lose the ability for happy accidents to evolve how our communication works? Is it really just all about the intent, which I think is important, especially when you're trying to retrieve information down the line? Or is it about the expression as well? So I'd say data is everything.

12:42And what I mean by that is all of this data is everything. You can't just take a little bit of it and pick and choose. You need to make sure that you're getting all of it and that we're not losing that nuance. We can always auto-tune it down the line. And we can use that to our benefit as cultural institutions. But we need to make sure that that's a choice. And I would also say that data is everything in terms of within the AI moment. We can use data as a point of leverage in that system to guide ai in places that we'd like it to go now we can't overplay our hand we can't change everything about the way that ai is constructed but i know that if we create good data and make good data sets available that those will work their way into the ground floor of the way ai is made in basically every model maker.

13:31And so if data is everything and we have leverage in that, what can we use it for? Well, at the Institutional Data Initiative, this is why we exist. It's not just to make better data. It's to use that to get something in society that we want. And I want to see cultural institutions as authoritative sources going forward. Historically, that's been the case. And to say it almost seems mundane, but it's at risk. And if we don't make sure that cultural institutions are the bedrock of certain types of information and AI systems, then we'll lose that place. But if we can secure it, then we can secure sustainable relationships in that ecosystem.

14:07This is something that I know that people at Wikimedia think a lot about. You want to make information available to everybody. And at the same time, you have certain players that are scraping the systems in ways that are diminishing the experience for standard users of that system. You have to find sustainable paths forward. And that sustainability is not just about making sure that you can balance a budget on a particular day. It's about making sure that, and this is my perspective, that there is dependency here. I don't just want cultural institutions to come to depend on AI, I want AI to come to depend on cultural institutions, and we have to use data leverage to get there.

14:47And the last thing I'd say is that I want to make sure that cultural institutions are viewed as experts in this. They've created the information science for cataloging information. They understand how something as simple as a category can be a political choice. And right now, across all of the model makers, there are people that are reinventing a lot of these techniques in this way of thinking. Some of them are coming around. Some of them are hiring librarians, for instance. but we need to make sure that uh that cultural institutions remain the experts that the processes that they've developed work their way into this new ecosystem uh and also that nobody knows how to apply these systems better to artifacts like a newspaper than the knowledge institutions themselves that's it thanks so much thank you very much uh does anyone have questions yes could you please pass it on thank you thank you so much for your presentation and thank you

16:00for the work that you're doing um i noticed on the hugging face page for the institutional books that right now Access is restricted to those who agree to use it only for non commercial purposes. And I would like to know whether there are plans to drop these conditions and what the the rationale behind this for restricting the usage of what I perceive is public domain works from the 19th and 18th century.

16:31Yep, great question. So we'd like to make these materials as accessible as possible. there's a lot that goes into this this is a complicated landscape so when you have a million books you have a lot to think about from a rights perspective we did a lot of work to make sure that clearances on this were rock solid but we have to think about how that situates at scale and how that works into the use of materials going forward so i can't get into some of the specifics around that other than to say that harvard really likes to cross our t's and dot our i's as we would say. The other part of it is I think that time is a factor that we can play with here. There are certain things that in knowledge institutions would be considered very short time frames, but for technology companies would be considered long.

17:22And if you can use time for, I think about three things. And this is when I talk about data leverage. I think data leverage should be used for three things. You don't have to use them for all three. You can pick and choose, but you can get consultation, accommodation, and compensation. And I think that we need to be smart about which of those we choose when we're publishing data. and part of what we think about is when we work with cultural groups for instance we want to make sure they understand in the ecosystem what they might be able to ask for and whether they want to make things openly available whether they want to honor some cultural traditions and prioritize certain groups so all that to say our goal is to make things as open as we can we're working to increase the openness of it certainly uh we try and be uh aggressive in the ways that we do that and then when we work with other groups you might find that in consultation with them we don't publish things that could otherwise be open because they want to engage with a community in a different

18:25way. Anyone else? You've talked about the importance of cultural institutions remaining the experts. I'm wondering if you have any thoughts about the role of non-experts within that ecosystem. Yeah, absolutely. I'm a non-expert myself. I have a degree in graphic design. I think that it's immensely important. I've been empowered by the open source community, by the open data community.

19:04um finding systems i think we're in a new moment uh for information so when we think about with the institutional data initiative a million books there's a lot that we want to engage with the open community on that but we're talking about at such a scale how can we um how can we engage and be mindful especially in an age when ai is being used that the ways in which that collection evolves are always accountable to the original source material. How can we basically scale our community practices when a million books just hits all of a sudden, rather than slowly is worked on by a community like Project Gutenberg is a great example. What a beautiful community project. How can we build scalable community systems to make sure that the things that we're bringing back into the project.

20:00Are always headed toward that accountability to the original source material. So there's that. When it comes to processes, I think that there's immense opportunity for non-experts to be applying those. It's happening all the time. In fact, that's part of why my point of valuing the experts is simply that the boulders rolling down the hill. and people of all types are giving it new momentum and i just want to make sure that the experts don't get left behind because that diligence and that thoroughness are assets that uh at the current speed that things are moving often mean that uh they're not at the at the vanguard nor is that where they should be any other question so you've shown us a circular diagram where you see a cyclical relationship between data and ai but to us in the audience and

21:07the wikimedia project that looks like a one-way arrow we're not seeing ai contributing useful content to our data sets and we don't want it to quite frankly we definitely don't see a place for regenerative AI, writing Wikipedia articles, creating images for Collins. And putting anything into Wikidata. So it seems like a one-way knowledge transfer. And given that, what leverage do you think we can exert on the AI companies? We don't seem to have any leverage, given that they don't seem to have any interest in respecting any licensing agreements or terms of service that any data set puts up.

21:48thank you well i'm sympathetic to that i don't i don't know that that's i don't know that all of that is true so uh what's that so um for instance i know that uh wikimedia has some relationships with uh the companies and is in discussion with them to uh to make sure that there is some flow back uh i'm sure that there's somebody here at wikimedia that could uh speak to that better than i could uh but i'm just gonna tee it up i think there's a ceo somewhere which is uh so you know i'll let that be discussed perhaps there's another session that that would be better but in terms of speaking for the the types of media that are on wiki uh that are on wikipedia and across wikimedia um you know when we think about things like um making images more searchable so So the next thing we'll do with the million books is extract all of the photos and illustrations

22:48that are in that, along with the captions. And for things that don't have a caption, we'll caption them. Because the images otherwise will be less searchable. Now, when we do things like this, and we use AI, we flag it as AI, and we make sure that that can be disentangled, that those two things don't get conflicted or confused. And I think that having processes like that are important. But I certainly think there's opportunity there. I also think that we have to be mindful of the leverage. I'll take your point that we're not at the optimal point of using our leverage. We're at the very beginning, and we should ask for more.

23:22So I take that. Any other question? Well, I'm doing my sprint today. I suspect that last question has provoked all the others, so I risk going off topic here. We're in a situation in particular in Europe at the moment that there are some arguably quite dubious efforts to shake down libraries and prevent them from allowing any use of their content in order to develop AI in a public interest way or otherwise.

23:59And there's a risk that when we talk about concepts like data leverage, these will end up being used by corporate rights holders who never wrote the stuff in the first place. They've got the rights because there's a certain business model that allows you to get to where you are. How do we make sure that we can design and what do models of data leverage of control look like that don't end up? Being used by major corporate rights holders who ultimately are just looking to their shareholders benefits, and actually work in ways that allow that expertise, allow those ethics, those values of heritage institutions to shine through.

24:36So I'll give you an example of a project that I think has been beneficial. Google Books. How many people here know Google Books? A lot of people know Google Books. the million books that we published were scanned by google that's work that um is there's a report by open future openfuture.edu about the history of the google books project there are trade-offs there for sure but if you look at it uh what a lot of librarians will say is that this is work that wouldn't have otherwise gotten done and depending on which institution you're talking about there are different sets of restrictions on that information now um in some of those contracts it says 15 years and so in fact right now it's been about 15 years since that project started i can tell you talking to that team they care immensely about the libraries that they work with i can tell you that uh they themselves in their own uh in their own company have had to continually

25:36be champions for that project and i also think that that the the the public is Going to see the fruit of that as a long term partnership. The most important thing is to make sure that restrictions like these are not long term. When I talk about time horizons, this is one of the things I'm talking about. I think there can be a balance. I think there's reason to be cautious. But I also know that the timing is really good on some of the previous investments that have been made by knowledge institutions and by companies.

26:12The other thing I'll say is I think a lot about the US, and one thing I'd like to see libraries in particular have is regulatory advantage, regulatory advantage. So when you're publishing public domain materials, the public domain in the US is a very squishy thing. There have been different eras of copyright, some of which are clearer than others. And so every time a library is publishing something in the public domain, they have to be really diligent about the way that they clear it.

26:45that diminishes, just like with fair use, it diminishes the extent that the public domain actually serves the public. If we were to make libraries, give libraries special privilege to publish things that they believe are in the public domain and then have reasonable takedowns when reported otherwise, rather than statutory damages that could bankrupt an organization, we could actually allow libraries to be primary sources of information in ways that industry never could be. And now you're giving them a place in society that is assured, and you're building a new basis for a relationship and industry that can't be replaced by a contract.

27:24And clap. Yeah, go ahead. What?

Source and licence

Recording: File:Keynote - Greg Leppert — GLAM Wiki Conference 2025.webm by Conbene (2025-10-30), via Wikimedia Commons. Licence: CC BY-SA 4.0. The transcript, summary and chapters on this page are shared under the same licence (CC BY-SA 4.0).

The transcript is machine speech-to-text by the TidyTools Audio & video transcriber; the summary, key points and chapters were written by Claude from that transcript. Both may contain errors: check the recording before quoting it.