AWOL Index

This publication systematically describes ancient-world information resources on the world-wide web. The bibliographic data presented herein has been programmatically extracted from the content of AWOL - The Ancient World Online (ISSN 2156-2253) and formatted in accordance with a structured data model. In continuous operation since 2009, AWOL is a blog authored by Charles E. Jones, Tombros Librarian for Classics and Humanities at the Pattee Library, Penn State University.

We extract information from AWOL about both top-level and subordinate resources. Subordinate resources are those we deduce to be "part of" another resource (e.g., a single issue of a journal or a sub-section of a website). Top-level resources are the opposite: those resources described by AWOL for which we have detected no containing/superior resource.

The latest data extraction was performed on 9 July 2015. At that time, our software successfully extracted 1,301 top-level and 50,704 subordinate resources. For 94% percent of the top-level resources it was able to extract a textual description substantively different from the resource title. Dates of individual source posts in AWOL vary between 2009 and 2015; therefore, content in The AWOL Index will only be as current as the original blog post was on the day of extraction.

Technical Considerations

Data Formats

The AWOL Index uses JavaScript Object Notation (JSON) to store the structured bibliographic records dervied from AWOL blog posts. The HTML pages and indexes are derived from these JSON files. See data model for information about the fields used in the JSON files.

Data Model

The records that isaw.awol produces conform to a flat, idiosyncratic data model informed by the DCMI Metadata Terms and the Metadata Object Description Schema (MODS) (among others). We chose to create our own data model in JSON rather than an established, standard format for several reasons:

the varied and idiosyncratic nature of the material being catalogued necessitates a format and schema that can be modified on the fly as new attributes of interest are identified
JSON is quick and easy to read and write from Python
JSON is terse, unambiguous, and easily read with any text editor
JSON is is well understood as a format by developers outside humanities and library software communities and readily used in web applications

We anticipate no difficulty in converting JSON to other formats for use in other systems.
See our Data Dictionary for definitions of all data fields.

Deduplication

Many resources are cited more than once in AWOL, but The AWOL Index should only have one record for each resource. Cases of such repetition occur both within a single blog post (e.g., when the title of a single thesis appears multiple times in a list, once under each of several categories) and accross blog posts (e.g., in a descriptive post about a single journal and in another post that lists a large number of journals on a particular topic). It is necessary, therefore, for isaw.awol to detect each repetition of a resource and judiciously to merge the data extracted in association with each.
Repetition is detected during a parse session by keeping a list of resource keys parsed previously during that session (see Record Naming for more information on resource keys). If a newly created resource key matches a key already stored in the list, isaw.awol reads the previous resource record in from the corresponding JSON file and then attempts to merge the newly parsed information into that record before saving it again.
Merge logic is conservative: best efforts are made to prevent discard of data from either record. Identical values, of course, are always retained in the result. When a particular field in one record is empty but in the other holds a value, the value is propagated. When the field is a complex data type (like a list of keywords), and the two records have differing values, the union set is propagated. Only when both records have empty fields, or conflicting simple values (e.g., if one contained "1923" in the year field and the other contained "1924"), is the merged output field left empty. In the latter case, an warning message is generated and noted by the operator subsequent investigation on the assumption that the parse logic in isaw.awol is not coping well with content or structure in a particular blog post.

Language Identification

AWOL blog posts do not generally contain a clear indication of the language of the resources being described; however, they do frequently contain descriptive text copied from the home page of the resource, and this often is in the language of the resource itself. The isaw.awol package feeds the contents of the resource title and description to Marco Lui's langid package in an attempt to classify the language in question. If langid.classify() returns a language identification with a confidence value greater than or equal to 0.98, the language code and confidence value are stored in the JSON record.

Provenance

In order to acknowledge the dependency of a given record on external resources, to attribute derived information to its original source(s), and to facilitate downstream attribution and assessment of validity, completeness, and timeliness, isaw.awol embeds provenance information in each AWOL Index record. We employ terms from two open vocabularies to express relationships between each AWOL Index record and one or more other sources. When we extract information from an AWOL blog post, two provenance entries are created. The first uses the "citesAsDataSource" term from the Citation Typing Ontology (CITO) to link the AWOL Index record to the Atom XML version of the blog post from which it was created using the Tag URI assigned to that Atom XML entry by the Blogger system. We also use the CITO term "citesAsMetadataDocument" to point at the original blog post itself by way of its URL.

When awol.index merges two records (see Deduplication), all provenance entries in the source records are copied into the output. In addition, a new provenance entry is made to document the fact of the merge using the "Combine" term from The Workflow Motif Ontology. In such a dictionary, the fields key is used to indicate those fields that were affected by the merge.

A regular example is shown above in the data dictionary; the following example demonstrates the result of a merge operation:

"provenance": [
    {
        "fields": [
            "authors", 
            "contributors", 
            "description", 
            "domain", 
            "editors", 
            "end_date", 
            "extent", 
            "form", 
            "frequency", 
            "identifiers", 
            "is_part_of", 
            "issuance", 
            "issue", 
            "issued_dates", 
            "keywords", 
            "languages", 
            "places", 
            "provenance", 
            "publishers", 
            "related_resources", 
            "responsibility", 
            "start_date", 
            "subordinate_resources", 
            "title", 
            "title_alternates", 
            "title_extended", 
            "type", 
            "url", 
            "url_alternates", 
            "volume", 
            "year", 
            "zenon_id", 
            "zotero_id"
        ], 
        "resource": "tag:blogger.com,1999:blog-116259103207720939.post-1897904554138962656", 
        "resource_date": "2010-05-25T09:07:46.422-04:00", 
        "term": "http://purl.org/spar/cito/citesAsDataSource", 
        "when": "2015-07-11T20:07:04.930037"
    }, 
    {
        "resource": "http://ancientworldonline.blogspot.com/2010/05/eastern-mediterranean-landscapes.html", 
        "resource_date": "2010-05-25T09:07:46.422-04:00", 
        "term": "http://purl.org/spar/cito/citesAsMetadataDocument", 
        "when": "2015-07-11T20:07:04.930053"
    }, 
    {
        "fields": [
            "authors", 
            "contributors", 
            "description", 
            "domain", 
            "editors", 
            "end_date", 
            "extent", 
            "form", 
            "frequency", 
            "identifiers", 
            "is_part_of", 
            "issuance", 
            "issue", 
            "issued_dates", 
            "keywords", 
            "languages", 
            "places", 
            "provenance", 
            "publishers", 
            "related_resources", 
            "responsibility", 
            "start_date", 
            "subordinate_resources", 
            "title", 
            "title_alternates", 
            "title_extended", 
            "type", 
            "url", 
            "url_alternates", 
            "volume", 
            "year", 
            "zenon_id", 
            "zotero_id"
        ], 
        "resource": "tag:blogger.com,1999:blog-116259103207720939.post-1897904554138962656", 
        "resource_date": "2010-05-25T09:07:46.422-04:00", 
        "term": "http://purl.org/spar/cito/citesAsDataSource", 
        "when": "2015-07-11T20:07:04.933826"
    }, 
    {
        "resource": "http://ancientworldonline.blogspot.com/2010/05/eastern-mediterranean-landscapes.html", 
        "resource_date": "2010-05-25T09:07:46.422-04:00", 
        "term": "http://purl.org/spar/cito/citesAsMetadataDocument", 
        "when": "2015-07-11T20:07:04.933839"
    }, 
    {
        "fields": [
            "subordinate_resources", 
            "resource_key", 
            "keywords", 
            "languages", 
            "description", 
            "is_part_of", 
            "identifiers"
        ], 
        "resource": "http://purl.org/net/wf-motifs#Combine", 
        "term": "http://purl.org/net/wf-motifs#hasWorkflowMotif", 
        "when": "2015-07-11T20:07:04.943916"
    }
],

Record Naming

Each record in the AWOL Index needs a unique identifier for retrieval, filenaming, and other tasks. We use a two-part "context and key" approach. Records are grouped by the hostname portion of the URL associated with the resource it describes (e.g. www.perseus.tufts.edu or trismegistos.org). Within each group, each record gets a unique "resource key" that identifies it in the context of that host domain. Ideally, we would use the "slug" portion of the resource URL (i.e., everything after the hostname portion), but this is impractical for several reasons, including the massive length of some URLs, the inclusion in many of application-related elements that confer little value as titles or filenames (e.g., "php"), and the appearance of characters inimical to or annoying to deal with in filenames. Therefore, isaw.awol uses the following heuristics to construct the resource key for each record:

seperate the slug from the host and protocol portions of the URL
if the URL has no slug (i.e., the resource being described is located at the top level of the specified host name), then re-use the domain name in the following steps as if it were the URL slug
in the slug, replace all slashes, spaces, periods (full stops), commas, and the like with hyphens
normalize repetitive characters
if the result is longer than 80 characters or contains ampersands, question marks or whitespace, calculate a hash value from the modified slug using the SHA1 algorithm.
The result, or the hexadecimal digest of the SHA1 hash (if necessary), is used as the value of the resource key.

Resource keys are not only used for filenaming, they are embedded in the records themsleves. They are also used to detect duplication (q.v.) during parsing.

Structured Data

AWOL blog posts are descriptive documents. In contrast, the individual entries in The AWOL Index are structured data records in which the discrete elements of description (metadata) have been separated into standard fields. None of the HTML tags in the original blog posts are carried over into The AWOL Index records. Original HTML structure and formatting is used extensively to facilitate parsing of the original blog posts, but these elements — which necessarily vary widely in the original content — are superseded by the uniform structure of the data model imposed by The AWOL Index.

Text Encoding and Normalization

All text in HTML files is UTF-8 encoded. All text in JSON files is also UTF-8 encoded; however, all characters therein whose Unicode code points fall outside the Basic Latin block (i.e., non-ASCII characters) have been escaped using the standard Python notation for "Unicode literals".
Content from AWOL is, in part, collected from a vast variety of web pages around the world. Consequently, the original text formatting, character encoding, and associated factors (e.g., Unicode Normalization Form) also vary. Jones is at pains when collecting and updating information to ensure that all text incorporated into the blog, regardless of original language, writing system, and encoding, displays in a readably consistent manner in modern web browsers. This effort helps ensure that the AWOL export file (in Atom format) that is parsed to produce The AWOL Index conveys all textual content using the UTF-8 encoding. The isaw.awol package takes further steps to detect and repair text whose character encoding or HTML structure is broken, deploying Leonard Richardson's Beautiful Soup Python library for the purpose.
Text is further normalized to facilitate consistent display, sorting, searching, and comparison in downstream and third-party processing. All character text is converted to Unicode Normalization Form C (NFC). The Python standard unicodedata library is used for this purpose. A number of character replacements are performed as well, including the conversion of:

all hyphen and dash variants (including non-breaking and "soft" hyphens) into simple hyphens,
left- and right-double quotation marks (and "Microsoft Smart Quotes") into simple double quotation marks,
the Unicode ellipsis character (\u2026) into a string of three simple periods (full stops), and
all continuous strings of one or more whitespace characters (including tabs, new lines, and non-breaking spaces) into a single simple space character.

isaw.awol sometimes adds periods (full stops) to text in place of HTML <br> tags when the text will be used as part of a longer description.

Data Dictionary

The following fields may be found in the JSON records produced by isaw.awol:

authors

A list (length: 0-n) of strings each containing an author's name.

Example:

"authors": [ "Weisenhorn, Markus" ]

contributors

A list (length: 0-n) of strings each containing a contributor's name (i.e., responsible individuals other than authors or editors. See authors for example.

description

A string of 0-n characters containing a prose description of the resource. May contain newline characters.

Example:

"description": "Acta Palaeobotanica. ISSN: 2082-0259 (electronic version). ISSN: 0001-6594 (printed version). Acta Palaeobotanicais an international journal publishing high quality contributions to palaeobotany and palynology. It is the only journal in Central and Eastern Europe focused on all fields of palaeobotanical and palynological investigations and publishes original palaeobotanical, palaeoecological, palaeophytogeographical, palynological, and archaeobotanical papers in addition to monographs, comprehensive review and discussion articles and book reviews. The journal is open to contributors from all over the world. It is published regularly with one volume per year each comprising two numbered parts, printed in June (No. 1) and in December (No. 2). The language of the journal is English. All manuscripts to be published in the journal are peer reviewed by at least two referees, and after acceptance of corrected manuscripts printing time is only approximately 6 months. Acta Palaeobotanicais now an open access journal and currently abstracts and full text of the articles in the PDF format beginning from volume 1 (1960) are freely accessible onwards here. The internet service also provides catalogues for volumes and supplements published since 1960 and includes information on ordering forms of printed copies."

domain

A string of n characters containing the primary domain name of the resource URL.

Example:

"domain": "anthropology.jp"

editors

A list (length: 0-n) of strings each containing an editor's name. See authors for example.

end_date

not currently used

extent

Experimental. A string describing the extent of the resource, corresponding generally in sense to MARC 300 subfield a, DCMI Terms "Extent", or MODS <extent>. Values are unconstrained and follow the original data uncritically.

form

Experimental. A string describing the form of the resource, corresponding generally in sense to MODS <form>. Values are unconstrained and follow the original data uncritically.

frequency

Experimental. A string describing the frequency of publication of the resource, corresponding generally in sense to MODS <frequency>. Values are unconstrained and follow the original data uncritically.

identifiers

A dictionary containing standard identifiers associated with the described resource. The strings "isbn", "issn", and "uri" may appear as keys in such a dictionary. Subordinate structure is as follows:

isbn and issn: a subordinate dictionary with possible keys "generic" and "electronic". Each such key corresponds to a list of 1 or more strings containing the identifier values.
uri: a subordinate list containing URI values as strings.

Examples:

"identifiers": {
    "isbn": {
        "generic": [
            "978-9-490604-0-04"
        ]
    }
},

"identifiers": {
    "issn": {
        "generic": [
            "0942-5659"
        ]
    }
},

"identifiers": {
    "issn": {
        "electronic": [
            "2051-9745"
        ]
    }
},

"identifiers": {
    "issn": {
        "electronic": [
            "2212-9758"
        ], 
        "generic": [
            "2212-974X"
        ]
    }
},

"identifiers": {
    "uri": [
        "http://www.numismaticadellostato.it/web/pns/notiziario"
    ]
},

is_part_of

A dictionary containing information about a higher-level resource related to the resource described in the record. This dictionary can contain the following keys:

title_full: a string containing the full title of the higher-level resource.
url: a string containing the URL of the higher-level resource.

Example:

"is_part_of": {
    "title_full": "Cuneiform Digital Library Bulletin", 
    "url": "http://cdli.ucla.edu/pubs/cdlb.html"
},

issuance

Experimental. A string describing the manner in which the resource is (was) issued, corresponding generally in sense to MODS <issuance>. Values are unconstrained and follow the original data uncritically.

issue

A string containing the issue number (vel sim) for a particular issue of a journal or similar serial resource. Values are unconstrained and follow the original data uncritically.

issued_dates

Experimental. A string containing an expression of a date range for the publication of a resource. Values are unconstrained and follow the original data uncritically.

keywords

A list of strings, each of which constitutes an AWOL keyword.

Example:

"keywords": [
    "Afghanistan", 
    "journal", 
    "newsletter", 
    "architecture", 
    "open access"
],

languages

A list of strings, each of which contains a language code indicating the presumed language of the resource. The language codes correspond to entries in the IANA Language Subtag Registry and conform to the guidance laid out in RFC 4646. See further, Language Identification, below.

places

Experimental. A list of strings, each containing the name of a place associated with the publication of the resource. This field corresponds generally in sense to MODS <place>. Values are unconstrained and follow the original data uncritically.

provenance

A list (length: 0-n) of dictionaries each containing an assertion about a discrete relationship between the record and an external resource. Each dictionary can contain the following keys:

fields: a list (length: 1-n) of strings indicating the data fields in the record affected by the relationship.
resource: a Uniform Resource Identifier (URI) for the external resource to which the record is related.
resource_date: a string containing a date/time associated with the external resource (issuance, publication, etc.), formatted in accordance with ISO 8601.
term: a string containing a URI for a term in a published vocabulary. Indicates the nature of the relationship between the record and the external resource.
when: a string containing the ISO 8601-formatted date/time value at which the relationship was created (e.g., the moment of derivation or merging).

Example:

"provenance": [
    {
        "fields": [
            "authors", 
            "contributors", 
            "description", 
            "domain", 
            "editors", 
            "end_date", 
            "extent", 
            "form", 
            "frequency", 
            "identifiers", 
            "is_part_of", 
            "issuance", 
            "issue", 
            "issued_dates", 
            "keywords", 
            "languages", 
            "places", 
            "provenance", 
            "publishers", 
            "related_resources", 
            "responsibility", 
            "start_date", 
            "subordinate_resources", 
            "title", 
            "title_alternates", 
            "title_extended", 
            "type", 
            "url", 
            "url_alternates", 
            "volume", 
            "year", 
            "zenon_id", 
            "zotero_id"
        ], 
        "resource": "tag:blogger.com,1999:blog-116259103207720939.post-6963043557368275385", 
        "resource_date": "2014-02-28T07:48:20.997-05:00", 
        "term": "http://purl.org/spar/cito/citesAsDataSource", 
        "when": "2015-07-11T20:13:38.313172"
    }, 
    {
        "resource": "http://ancientworldonline.blogspot.com/2012/11/open-access-journal-abstracta-iranica.html", 
        "resource_date": "2014-02-28T07:48:20.997-05:00", 
        "term": "http://purl.org/spar/cito/citesAsMetadataDocument", 
        "when": "2015-07-11T20:13:38.313189"
    }
],

publishers

Experimental. A list of strings, each containing the names of publishers associated with a resource. This field corresponds generally in sense to MODS <publisher>. Values are unconstrained and follow the original data uncritically.

related_resources

not currently used

resource_key

A string of 1-n characters containing a unique identifier for the record in the context of the associated resource hostname. May not contain whitespace or newline characters. See further Record Naming, below.

Examples:

"resource_key": "abstractairanica-revues-org"
"resource_key": "LuwianCorpus-search"
"resource_key": "df06ecac37cbb6cd8695b618fe85709e21311cb9"

responsibility

not currently used

start_date

A string containing an indication of the start date of a serial or continuing resource. Format is unconstrained; values will match whatever formatting is found in the original data.

subordinate_resources

A list of dictionaries, each of which uses the same conventions outlined for is_part_of to identity a resource that is seen to be subordinate to the one described by the record.

Example:

"subordinate_resources": [
    {
        "title_full": "Latest Issue of Ancient TL", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/06-12/"
    }, 
    {
        "title_full": "Previous Journal Issues", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/"
    }, 
    {
        "title_full": "Volume 31, No 1. - Jun 2013", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/06-13/"
    }, 
    {
        "title_full": "Volume 30, No 2. - Dec 2012", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/12-12/"
    }, 
    {
        "title_full": "Volume 29, No 2. - Dec 2011", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/12-11/"
    }, 
    {
        "title_full": "Volume 29, No 1. - June 2011", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/06-11/"
    }, 
    {
        "title_full": "Volume 28, No 2. - Dec 2010", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/12-10/"
    }, 
    {
        "title_full": "Volume 28, No 1. - Jun 2010", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/06-10/"
    }, 
    {
        "title_full": "Volume 27, No 2. - Dec 2009", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/12-09/"
    }, 
    {
        "title_full": "Volume 27, No 1. - Jun 2009", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/06-09/"
    }, 
    {
        "title_full": "Volume 26, No 2. - Dec 2008", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/12-08/"
    }, 
    {
        "title_full": "Volume 26, No 1. - Jun 2008", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/06-08/"
    }, 
    {
        "title_full": "Volume 25, No 2. - Dec 2007", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/12-07/"
    }, 
    {
        "title_full": "Volume 25, No 1. - Jun 2007", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/06-07/"
    }, 
    {
        "title_full": "Volume 24, No 2. - Dec 2006", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/12-06/"
    }, 
    {
        "title_full": "Volume 24, No 1. - Jun 2006", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/06-06/"
    }, 
    {
        "title_full": "Volume 23, No 2. - Dec 2005", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/12-05/"
    }, 
    {
        "title_full": "Volume 23, No 1. - Jun 2005", 
        "url": "http://www.aber.ac.uk/en/iges/research-groups/quaternary/luminescence-research-laboratory/ancient-tl/journal/06-05/"
    }, 
    {
        "title_full": "here", 
        "url": "http://www.aber.ac.uk/temp-ancient-tl/"
    }
],

title

A string containing the title of the resource.

Example:

"title": "R\u00f6mische Hinterlassenschaften im Gemeindegebiet Kladovo in Serbien",

title_alternates

not currently used

title_extended

Experimental. Contents are undefined.

type

Experimental. A string that describes the type of the resource. This field corresponds generally in sense to MODS <typeOfResource>. Values are unconstrained and follow the original data uncritically.

url

A string containing the Uniform Resource Locator (URL) of the resource described by the record.

Example:

"url": "http://othes.univie.ac.at/25727/",

url_alternates

not currently used

volume

A string containing the volume number (vel sim) for a particular issue of a journal or similar serial resource. Values are unconstrained and follow the original data uncritically.

year

A string containing the year for a particular issue of a journal or similar serial resource. Values are unconstrained and follow the original data uncritically.

zenon_id

not currently used

zotero_id

not currently used

The AWOL Index

Table of Contents

Funding for the preparation of The AWOL Index has been provided by:

Contact

Copyright and Licensing

How to cite The AWOL Index

Technical Considerations

Data Dictionary