Data Formats

• RDF - Triples (XML, Turtle, N3 serializations
• MODS, MARC, Dublin Core, XML format
• CSV - Comma Separated Value
• JSON - Feeds both incoming and outgoing

Datastores

• Triplestore - Stores RDF
• SQL DBs - Relational table structures
• Repositories - Store digital objects
• NOSQL- key-value, document, types
• Filesystems - Store digital files and cached resources

Query Languages

• SPARQL - for RDF Triplestores
• SQL - Relational DBs
• XPath for XML document processing/query
• DSL - an example of a NOSQL query language (Elasticsearch)

Python Modules

• Flask - Build HTML5 Applications using Bootstrap, JQuery
• RDFLib
• lxml
• Pymarc
• bibcat

Open-Source Enterprise Tools

• Triplestore - Blazegraph, rdflib
• Digital Object Repository - Fedora 4
• Search index - Elasticsearch
• Filesystem - Linux, MacOS, and Windows

Source Control, Packaging, and Deployment

• Git and Github
• Python Module Repository PyPi
• Docker

Data Sources

MARC 21 and MARC XML records are exported from an ILS

MODS XML, OAI-PMH Dublin Core, Fedora Commons XML, are harvested from OAI-PMH Ingesters typically from external repositories like Islandora, Omeka, or Content DM digital repositories.

Excel and Comma-Separate-Value, in a couple of instances, the data source are spreadsheets that either start as Microsoft Excel that are converted to CSV, or the original source is CSV.

JSON feeds, for some projects, a JSON feed of Dublin Core metadata is the record source

SPARQL endpoints

Application Programming Interfaces API

BIBCAT Ingesters

1. Data harvested over OAI-PMH endpoints, HTTP connections, or provided data files
2. Usually executed from the command-line and can be long-running processes

BIBCAT RML Processors

BIBCAT uses RDF Mapping Language to preform the following processing of incoming data to the final RDF Linked Data output, stored either in triplestore or on disk

• Simplification - Library of Congress BIBFRAME RDF to Lean BIBFRAME RDF for production efficiency
• Normalization, try to match new RDF entities with existing entities ether locally or through external sources like LOC name and subject authorities, FAST, or ORCID
• Standardization - minting IRIs that follow certain patterns, like a title and author slug in BIBFRAME Instance and Items IRIs
• Mappings - Sometimes complex mapping logic between incoming data and production RDF
• Indexing/Search - flatten and expand RDF entities to be indexed into search engine

BIBCAT uses the Rdfframework for configuration and data management

Access & Discovery

Using RML Processors, BIBCAT is easily able to generate multiple formats and vocabularies outputs depending on the project.

• Search Engine indexing by Google and Microsoft Bing using schema.org JSON-LD for the Colorado Alliance's GoldRush BIBCAT project.
• DP.LA Service Hubs, BIBCAT is used in the regional service hub for Colorado and Wyoming, generates a ResourceSync XML feed linking MAPv4 JSON-LD for inclusion in DP.LA
• JSON API for Kean University's Academic Concierge iOS App
• Web front-end user interface - currently being actively developed as BIBCAT Publisher - projected release second quarter 2018