A Python Interface for api.Biblia.com

Last week Logos announced a public API for their new website, Biblia.com, at BibleTech. Of course, i want to wave the flag for my employer. But i’m also interested as somebody who’s dabbled in Bible web services in the past, most notably the excellent ESV Bible web service (many aspects of which are mirrored in the Biblia API: some previous posts around this can be found here at Blogos in the Web Services category). Dabblers like me often face a perennial problem: the translations people most want to read are typically not the most accessible via API, or have various other limitations.

So i’m happy with the other announcement from BibleTech last week: Logos is making the Lexham English Bible available under very generous terms (details here). The LEB is in the family of “essentially literal” translations, which makes it a good choice for tasks where the precise wording matters. And the LEB is available through the API (unlike most other versions you’re likely to want, at least until we resolve some other licensing issues).

I don’t want to do a review of the entire API here (and it will probably continue to evolve). But here are a couple of things about it that excite me:

  • The most obvious one is the ability to retrieve Bible text given a reference (the content service). Of the currently available Bible versions, the LEB is the one that interests me the most here (i hope we’ll have others in the future).
  • Another exciting aspect for me is the tag service. You provide text which may include Bible references: the service identifies any references embedded in it, and then inserts hyperlinks for them to enrich the text. So this is like RefTagger on demand (not just embedded in your website template). You can also supply a URL and tag the text that’s retrieved from it. One caveat with this latter functionality: if you want to run this on HTML, you should plan to do some pre-processing first, rather than treating it all as one big string. Otherwise random things (like “XHTML 1.0” in a DOCTYPE declaration) wind up getting tagged in strange ways (like <a href="http://ref.ly/Mal1">ML 1.0</a>).

I’ve just started working through the Biblia API today, but since i’m a Pythonista, developing a Python interface seemed like the way to go. This is still very much a work in progress, but you can download the code from this zip file and give it a whirl. Caveats abound:

  • I’ve only implemented three of the services so far: content() (retrieves Bible content for a reference), find() (lists available Bibles and metadata), and tag() (finds references in  text and enhances it with hyperlinks). And even with these three services, i haven’t supported all the parameters (maybe i will, maybe i won’t).
  • This is my first stab at creating a Python interface to an API, so there may be many stylistic shortcomings.
  • Testing has also gotten very little attention, and bugs doubtless remain.

If you’re interested and want to play along, let me know: we can probably set up a Google group or something for those who want to improve this code further.

Bookmarklets Redux

Time spent on the web can be oh-so tedious if you’re constantly cutting things from one page and pasting them elsewhere just to get to another, related page. Someday Linked Data may make this all better, but until then, we all get by with helpful tricks.

Bookmarklets are one essential weapon in the arsenal of the web-info-warrior. Usually they’re little JavaScript programs stored as a bookmark in Firefox, providing one-click access to some simple functionality like looking things up elsewhere, resizing your window, etc. I’ve blogged previously about bookmarklets to find local library sources for a book on an Amazon page (or PaperBackSwap).

I dusted off my bookmarklet skills this past week and came up with some nifty tools that i wanted to share.

First off, imagine you’re looking at a website with Bible references whose benighted author somehow failed to include RefTagger. So rather than a nice pop-up with the text of the reference, or even a helpful link to that text on some Bible site, you’re just looking at a inanimate, unlinked string: boo. The Bible Reference Bookmarklet to the rescue! Simply select the text of the reference, click the bookmarklet, and you’ll be whisked off to that reference at Bible.Logos.com. If you haven’t selected any text first, you get a dialogue box asking for it.

To get this goodie in Firefox, first make sure the Bookmarks Toolbar is showing (View > Toolsbars > Bookmarks Toolbar must be checked). I’d love to give you a link to just drag onto the toolbar, but i don’t seem to be able to get the code past WordPress. So go to Bookmarks > Organize Bookmarks, and select Organize > New Bookmark. Give it a useful name like “Bible Reference Lookup”, and paste the code below in Location field.

javascript:(function(){%20function%20getSearchString%20(promptString)%20{%20s%20=%20null;
if%20(document.selection%20&&%20document.selection.createRange)%20{%20s%20=document.selection.createRange().text;%20}%20
else%20if%20(document.getSelection)%20{%20s=%20document.getSelection();%20}%20
if%20(!%20(s%20&&%20s.length))%20{%20s%20=prompt(promptString,'');%20}
%20return%20s;%20}%20searchString%20=%20getSearchString('Bible%20Reference%20to%20look%20up%20:');%20
if%20(searchString%20!=%20null)%20{%20if(searchString.length)%20{%20location%20='http://bible.logos.com/#ref='+escape(searchString);%20}%20
else%20{%20location%20='http://bible.logos.com/';%20}%20}%20%20})();

After you’ve clicked ok, you should see it on your toolbar.

You can do similar tricks for a wide variety of strings that you just want to look up elsewhere (i discovered one here while writing this post that lets you look up articles on Wikipedia). This isn’t fundamentally different from copying the string into a search box: but sometimes it’s more convenient.

Descending into more esoteric purposes (to give you ideas for your own bookmarklets): as part of an earlier post on Tools for Personal Knowledge Management, i mentioned my use of TiddlyWiki for quick organization of hyperlinked notes. Like other wiki software, TiddlyWiki has its own link syntax, that looks like

[[Link text | URL]]

When linking to lots of other web pages, i was getting tired of copying the URL, pasting that in, then typing the square brackets, link text, vertical bar, and more square brackets, all in the right format. Wouldn’t it be more convenient to just construct this expression from the title of the page and its URL, rather than having to type it myself? YES! and the TiddlyWiki Page Link bookmarklet does just that, putting the result in a little pop-up window where a triple-click selects the whole thing, ready to copy and paste into your tiddlywiki (and tailor as desired: the title isn’t always what you want, but it’s often easier to edit and throw things out rather than type afresh). This one you can just drag to your bookmarks toolbar and use right away.

TiddlyWiki Page Link

Also, i’ve switched to a much better library lookup bookmarklet (and a service to help you create one for your local library) from WorldCat. Among other things, it generates the list of all the different ISBNs that might exist for a title (which can be very long indeed), and when there are many, it provides links for alternate searches in case the first group comes up empty handed.

Some other cool bookmarklets in my collection include:

  • CiteULike Popup Post and kin to make it easy to add (certain kinds of) articles to your reading list management. Adds more value for sources whose structure it understands.
  • Show del.icio.us citations of the current URL (you can find it there)
  • Resize your browser window to 1024 x 768 (if you want to see how a page will look on a smaller monitor or projector): the bookmarklet follows, just drag to your toolbar. 1024 x 768
  • A CSS validator for the current page: see Pete Freitag’s page.

Hat tips:

xpound.org and Web 2.0 Bible tagging

xpound.org is a new Web 2.0 site that provides passage search, blogging, and social connections, but with an interesting new twist of Bible tagging, along the lines of del.icio.us. (I’m not sure if it’s pronounced with equal stress like “slashdot”, or “ex-POUND”, like the verb) The basic idea of tagging is that, rather than a top-down, authoritative organization and labeling of knowledge, people can simply attach whatever labels make sense to them, in a bottom-up, unstructured (and, some would say, chaotic) fashion. The natural advantage of this kind of folksonomic tagging is that, at internet scale, it can overcome a lot of the messiness, while highly structured knowledge management approaches don’t always scale. As with other tagging sites, there’s no guarantee that what somebody tags as, say, africa, will have meaning to anybody else. But it means something to the person who tagged it, and thus becomes a highly personalized way to organize information.

I think using this approach for Scripture makes some sense, and i’ve blogged about it previously. But i also have some questions. With del.icio.us, the item being tagged is clearly defined: it’s a URL. But what’s the natural unit for tagging Scripture? Verses are one answer, but they often don’t have enough context. Books are generally too large, and chapter divisions don’t necessarily line up with the content you’d want to tag. Of course, you can tag arbitrary passages: but here’s where the comparison to del.icio.us breaks down. With del.icio.us, others who use the same tags as me can point me to sites i didn’t know about. But where the passages aren’t necessarily bounded the same, aggregation doesn’t work quite the same way.

Here’s an example: a few days ago Josh tagged Ephesians 5:3-4 with five tags: gratitude, greed, immorality, impurity, and saint.

3 But sexual immorality and all impurity or covetousness must not even be named among you, as is proper among saints. 4 Let there be no filthiness nor foolish talk nor crude joking, which are out of place, but instead let there be thanksgiving. (Eph.5.3-4)

If you search on these tags (like gratitude), you’ll find this unit. If instead you search by verse for Eph5.3, you still see the tag for gratitude, presumably inherited from the larger unit that was tagged (though you could argue that gratitude really only applied to verse 4). And of course, the following verses also talk about immorality and impurity, though (since they weren’t included in these tags) they’re not retrieved.

One of the most interesting new capabilities that del.icio.us creates is knowledge discovery: if i find someone who has bookmarked several of the same sites as i have, i can go look at their other bookmarks, and often find new sites i was unaware of. This provides a kind of search by likeminded community intelligence, a really interesting counterpart to typical web search engines. I haven’t found this capability in xpound, but it would be a great addition.

(Hat tip to the ESV Blog for pointing me to xpound.org)

Google Homepage Daily Bible Verse

Here’s a fun little hack. For a while now, Google has offered people a way to customize their homepage (to do this, you need a Google account). This allows you to add things like RSS feeds by URL, or certain other bits (like weather, quote of the day, or Gmail) that Google has pre-defined.

They recently released an API that enables you to customize the content even further by creating (in Google parlance) a “gadget”: for example, you can parse an RSS feed and format it yourself in HTML. This is especially useful for Daily Verse feeds: i don’t generally want to see a reference to a verse, i want the verse content itself.

So i’ve put together a Google gadget that parses the ESV’s Daily Verse RSS feed and displays the verse content, along with the reference (hyperlinked to the ESV site). To get it (assuming you have a Google homepage),

  1. Click on the “Add Content” link on the upper left
  2. On the next page, click on the “Add by URL” link (to the right of the “Search Homepage Content” button)
  3. Paste http://www.semanticbible.com/blogos/gems/dailyverse.xml into the box and click Add

 

Hyper-Concordance and the ESV Web Service

I spent some time last weekend working on changing the hyper-concordance to a MySql backend. The current implementation simply generates a large number of static HTML files: easy to implement, but a pain to move that much data around. Since there’s a file for each term, that’s about 3000 files, and 30+ Mb of data. Worse, each verse is repeated for each of its indexed terms:

“For the kingdom of heaven is like a householder who went out early in the morning to hire laborers for his vineyard.(Matt.20.1, RSV)

winds up being stored nine times.

The obviously superior approach, unimplemented not because i’m stupid but because i’m lazy, is to put each verse in a database, create an index of terms to verses, and then serve pages that are generated dynamically and styled on the fly.

But my recent ruminations on Web 2.0 buzz got me thinking that it might be time to try building on the ESV Web Service API instead. Here’s an outline of my thinking:

  • Use the same perl code i already have to map inflected terms back to their bases (more about this here: by the way, this is the only thing that seems even modestly new to me about the hyper-concordance)
  • a term request gets mapped into a series of verse requests using doPassageQuery (note i can’t use doQuery: that would defeat the mapping back to base forms). Looks like you can retrieve multiple passages by specifying something like “matt.15.23, matt.15.24, matt.15.32, matt.15.39” as the reference (four verses from Matt 15 with different forms of “send”).
  • the resulting XML gets tokenized to identify the base terms, and the processed to add in the hyperlinks and some CSS styling (for example, bolding the query term)
  • the results get sent to the browser

Some remaining practical questions:

  • the largest entries in my (yet unreleased) ESV index have hundreds of verses (i think “say” is the current winner). This can be reduced by adding more things to the stopword list, but only at the cost of losing them as hyper-terms. Will the API hold up when queries reference this many passages?
  • Is this acceptably fast?
  • is this well-behaved enough for the daily limit of 500 queries? I don’t think i have that many users based on server logs, but i would be nice to be scalable

I’m looking forward to experimenting with this approach: stay tuned.

More Fun with Firefox and ESV Search

I keep finding new features i like in Firefox. The latest is keyword search in the “location bar” (where you normally type a URL). Type “weather” there and you get weather.com. Type “stock vz” and you get a quote for Verizon stock (please buy some and help get my options above water!).

The behavior is a little hard to predict, though. Entering “time” comes up empty-handed (“The operation timed out when attempting to contact time”: pun unintended i’m sure), but “time now” brings up this page with (you guessed it) the current time. It seems to be hit or miss: “bible” brings up Bible Gateway, “new testament” brings up Goodacre’s blog, “concordance” brings up this page with interfaces for searching Strong’s. “semanticbible” (no spaces) brings up my site, with a space “semantic bible” it brings up Glenn’s reference to my site,  but “semantic” itself brings up nothing. I’d love to have a real description of what’s happening behind the scenes.

The cool thing is you can define your own. Here’s how i set up a passage search against the ESV:

  • Ctrl-B to view the bookmarks in the sidebar
  • right-click on Quick Searches to create a New Bookmark
  • Name = “ESV Passage Search”
  • Location = http://www.gnpcb.org/esv/search/?passage=%s
  • Keyword = “esv”
  • Building on their existing examples, i put ‘Type “esv <ref>” to ook up passages in the English Standard Version Bible’ for the description

Now you can go to the location bar, type “esv Col 3:16”, and get the verse. The capability is fully general: whatever terms you supply get substituted for the %s in the location URL. Items in your search bar also have this capability (e.g. you can do “amazon <bookname>” and search that way).

Alas, entering “blogos” as a quick search doesn’t bring up my blog as it should, but instead that upstart blog on language and technology that appropriated my name a full year after i’d been using it (me, bitter? never! ).

ESV Text for Blogging Via Webservice

I haven’t posted an ESV webservice hack in a while, but since i’m on vacation, i had some time for recreational programming, and whipped up something i’ve wanted for a while: a simple program to take one or more verse references, retrieve the results from the ESV web service, and reformat them slightly to make it easier to compose blog posts. Here’s the full story.

Is it crazy to program while on vacation? No! I find it energizing and enjoyable, and i don’t usually have enough time for things like this.

Scripture Indexes in OPML

OPML (Outline Processor Markup Language) is an XML format designed by Dave Winer of Radio Userland. It’s a lightweight XML structure meant for simple outlines that promotes sharing and hyperlinking. The Radio Userland weblogging tool also incorporates some nice OPML features, like an outline-based editor and automatic rendering and publishing mechanisms.

I wrote an XSL transformation to convert my Scripture indexes into an OPML format, sans the text of the verse itself (but hyperlinked to the ESV site). For the imperatives of Jesus from Luke’s Gospel, here’s the OPML version that corresponds to this XML version.

This is really only relevant if you already have some OPML tools at hand: otherwise it’s just another format for the same information. I’m still learning about the benefits of OPML, but (given appropriate tools) it allows others to “subscribe” to these files. Since things always change, that might be a benefit. It doesn’t directly expose all the attributes in the Radio Userland tool (like the scope, whether it’s implicit, or who the imperative was directed to), though you can find them through a right-click and selecting Debug (though that seems like a curious interface). But like i said, i’m still learning…