BibleTech talk: Automatically Learning Topical Content

I’ve posted the slides from my BibleTech 2013 talk. Here’s the abstract:

Continued work on the Logos Controlled Vocabulary (BibleTech 2010, “A Controlled Vocabulary for Biblical Studies”) has produced a unique collection of topic-aligned content across more than 50 different Bible dictionaries, encyclopedias, and topical indexes in both English and Spanish. This presentation will describe the information we’re learning automatically from this content, including:

  • determining concept importance
  • associating concepts with Bible references
  • extracting and associating names and descriptive terms for concepts
  • relating concepts to each other

You can see the other talks at the BibleTech website. I’ve had a number of positive comments on the talk, which is always gratifying. Slowly but surely, we’re climbing up the data stack …

3/21/2013 update: I’ve used the Slidy framework for presentations for several years because i like the way it puts the whole content out on the web in HTML. However, @JohnRGentry pointed out that my slides don’t work on the iPad because moving forward and backward requires keys. There’s a newer version of Slidy which does support swiping to move through slides, though my experience with it on both Safari and Chrome on iOS hasn’t been great: it’s not easy to register a swipe, and title text sometimes gets lost. I assume these are issues with javascript support on iOS, though i’m not really sure. I’ll try to update my slides to the newest version of Slidy, which will help a little, but i’ll also look for another framework that’s more tablet friendly.

Logos 5, Behind the Curtain: Curation

This post is part of Logos 5, Behind the Curtain, a series of blog posts looking at new data sets that are part of the latest Bible study application from Logos Bible Software.

At the heart of Logos’ approach to data is the practice of curation. If you’ve heard this term at all, it’s probably in the context of museums: but we mean something rather different. As usual, Scott Adams’ Dilbert has the pulse of corporate America:

The Official Dilbert Website featuring Scott Adams Dilbert strips, animations and more

For Logos, though, curation is not just trendy new jargon, but an essential practice that we’ve been pursing for quite a few years now (five under my direction, and more before my tenure). It’s a critical part of what makes Logos unique in its market. For example, we have not one but three Lexical Curators working on the Bible Sense Lexicon (one for Greek, two for Hebrew). To my knowledge (and Google’s), these appear to be the only people in the world with this job title. In fact, most people in the Content Innovation department at Logos are doing one kind of curation task or another.

So what is curation in the context of Bible study software, anyway? In the simplest terms, curation means organizing and maintaining a collection of things. In the museum context, that’s artifacts they display. Our kind of curation involves computer-readable data that captures knowledge relevant to biblical studies. You can summarize it briefly with three key practices:

  1. Collect
  2. Correct
  3. Connect

(“Describe” is probably a more apt term than “Correct”, but i just couldn’t resist the awesome alliteration.)

To “Collect” means imposing organization on a complicated and messy world of information, with an eye toward structuring it and making it useful in some way in the Logos system. Often this information is expressed in prose sentences in a reference book: for example, a Bible dictionary might describe a city, and include language about where the city was located, what larger region it was part of, where it’s mentioned in the Bible, etc. Fundamentally, a plan for collection requires deciding what information ought to be collected, and which things belong in the collection and which ones are excluded (in more technical terms, an ontology).

Capturing and formalizing knowledge typically involves some tradeoffs. First of all, we have to find categories and labels that balance and maximize utility and (formal) correctness. When it comes to categorization in particular, ultimately “everything is miscellaneous” (which is also the title of a really important book by David Weinberger), and if you push hard enough, each thing is its own unique category . But data sets are typically more useful if things are grouped in some way. So we categorize the following as “people” in our data set:

  • individuals (whether named or not)
  • groups, whether defined by residence (Greeks), common ancestry (Levites), belief systems (Pharisees and Sadducees) , etc.
  • supernatural entities (including those that most Bible readers would accept like the God of Israel, and those that biblical authors argue are not real at all, like Baal, Asherah, or Zeus)

To “Correct” or “Describe” means that we choose and populate particular attributes for the items we’re curating. In the case of people, that includes things like names, gender, and roles. In addition, we create three special attributes for nearly every data set:

  • a unique identifier: though you’ll probably know who i mean when i use a biblical name like “David”, you won’t for “John” because there are five different individuals known by that name. This kind of ambiguity, and other variations (“John”, “John the Baptist”, “John Baptizer”, etc.) means names are usually poor identifiers. Instead, we create a symbol like “John.4” that uniquely identifies one particular item in our collection (in this case, a member of the Sanhedrin mentioned in Acts 4:6). Since we don’t show these to users, we don’t have to worry about people understanding them.
  • a label: since our data is for people to look at and understand, we need user-friendly ways to display an entity. Labels are typically brief (less than 20 -25 characters), and also unique, so that in a drop-down list showing names that match “John”, i can distinguish “John (the Baptist)” from “John (Ac 4:6)”. Since the label is also unique, we could use it as the identifier: but since data stability is a primary goal, we separate the two, so that we can change the label if necessary. Since identifiers (not labels) are the means by which we connect data, we (almost) never change them, since that risks breaking the integrity of the data set.
  • a description: labels are brief so they take up minimum space, but consequently, they can’t carry much information. So we often provide a longer prose description, perhaps a sentence or two, that helps identify the entity and its most basic information. You could compare this to the leading sentence in a Wikipedia article. In the case of John.4, that’s “a member of the family of high priests in Jerusalem following Jesus’ ascension.”, which is probably enough to help you decide whether this is a John you want to know more about or not.

To “Connect” means linking entities to other entities (or other data sets). For people, family relationships are an important ways that people connect to each other. We also label those relationship (father, mother, sister), and, for biblical information, capture the textual sources that support this relationship (more technically, the provenance of the data).

Connecting information is one of the most important aspects of curation for Logos. While it may be interesting to learn that King David was also a shepherd, that’s an isolated fact. But if you can get a list of other individuals in the Bible who were also shepherds (or kings, or musicians), now you’re discovering new information. You might not have started out looking for this, or known how to find it for yourself.

Question: which part of Logos’ curation process (Collect, Correct, Connect) do you find the most interesting or appealing? Please leave me a comment.

(Edit: saw a good piece today by John Chambers of Cisco about the power of connection.

Logos 5, Behind the Curtain: Introduction

I’ve fallen off in my blogging quite a bit over the last few years: my last post (a book review) was December 2011, and, other than conference reports and book reviews, it’s rather sparse for the year or two prior to that.

But i’m excited to be reviving my blog and starting a new series today to celebrate the release of Logos 5. In Bob’s Pritchett’s overview post on the Logos blog, he talks about the importance of connection in Bible study, and says [italics mine]

Logos Bible Software 5 is a significant update that is all about connection.

This focus on connection is not just marketing talk or a conceptual metaphor. It describes in a very concrete way the important new datasets that make Logos 5 a major contribution to the world of biblical studies. I recognize that’s a mighty big claim, but i plan to back it up by describing the vision, architectures, effort, and technical approaches that have made the new features of Logos 5 a reality. In my role as Director of Content Innovation at Logos, i lead a talented and hard-working team of people who have spent the last several years making all these connections. (it wasn’t my idea, but that’s me in the “What’s new in Logos 5” video) Over this series of posts, i hope you’ll gain a better appreciation for what went into these new features, and what makes them so important.

I’d love to hear your comments and questions. For example: which of the new Logos 5 features do you find most useful?

A Python Interface for

Last week Logos announced a public API for their new website,, at BibleTech. Of course, i want to wave the flag for my employer. But i’m also interested as somebody who’s dabbled in Bible web services in the past, most notably the excellent ESV Bible web service (many aspects of which are mirrored in the Biblia API: some previous posts around this can be found here at Blogos in the Web Services category). Dabblers like me often face a perennial problem: the translations people most want to read are typically not the most accessible via API, or have various other limitations.

So i’m happy with the other announcement from BibleTech last week: Logos is making the Lexham English Bible available under very generous terms (details here). The LEB is in the family of “essentially literal” translations, which makes it a good choice for tasks where the precise wording matters. And the LEB is available through the API (unlike most other versions you’re likely to want, at least until we resolve some other licensing issues).

I don’t want to do a review of the entire API here (and it will probably continue to evolve). But here are a couple of things about it that excite me:

  • The most obvious one is the ability to retrieve Bible text given a reference (the content service). Of the currently available Bible versions, the LEB is the one that interests me the most here (i hope we’ll have others in the future).
  • Another exciting aspect for me is the tag service. You provide text which may include Bible references: the service identifies any references embedded in it, and then inserts hyperlinks for them to enrich the text. So this is like RefTagger on demand (not just embedded in your website template). You can also supply a URL and tag the text that’s retrieved from it. One caveat with this latter functionality: if you want to run this on HTML, you should plan to do some pre-processing first, rather than treating it all as one big string. Otherwise random things (like “XHTML 1.0” in a DOCTYPE declaration) wind up getting tagged in strange ways (like <a href="">ML 1.0</a>).

I’ve just started working through the Biblia API today, but since i’m a Pythonista, developing a Python interface seemed like the way to go. This is still very much a work in progress, but you can download the code from this zip file and give it a whirl. Caveats abound:

  • I’ve only implemented three of the services so far: content() (retrieves Bible content for a reference), find() (lists available Bibles and metadata), and tag() (finds references in  text and enhances it with hyperlinks). And even with these three services, i haven’t supported all the parameters (maybe i will, maybe i won’t).
  • This is my first stab at creating a Python interface to an API, so there may be many stylistic shortcomings.
  • Testing has also gotten very little attention, and bugs doubtless remain.

If you’re interested and want to play along, let me know: we can probably set up a Google group or something for those who want to improve this code further.

Bible Data Visualization Blog

camaris has started a Bible Data Visualization blog to practice some visualizations. The goal:

… show 40 visualizations of the Holy Bible. Most of the visualizations will be self-made, but sometimes I will cover the work from other people.

Looks like there’s also some narration of the process, which may be useful if you’re thinking about how to do some visualizations yourself.