citoid

Citoid in VisualEditor

The citoid node.js service generates citation data given a URL, DOI, PMID, or PMCID. It has a companion extension, Citoid, which aims to provide use of the citoid service to VisualEditor. It is currently deployed in all VisualEditor-enabled WMF-Wikis, though the extension is only configured on a few of them.

Public API

Main article: Citoid/API

To request metadata about a URL, DOI, PMCID or PMID, you can use the API endpoint at citoid.wikimedia.org and provide a GET request with the required parameters "search" and "format".

The "search" parameter takes as value the URL-encoded URL, DOI, PMCID or PMID.

The "format" parameter takes the values "mediawiki", "bibtex", "zotero", and "mwDeprecated".

An example request is http://citoid.wikimedia.org/api?format=mediawiki&search=http%3A%2F%2Flink.springer.com%2Fchapter%2F10.1007%2F11926078_68

Issue tracker and project management

Bugs, issues, and suggestions for improvement can be added to the Citoid phabricator project.

Installation

Best results are obtained if the URLs that are popular on your site are already available in Zotero. If they're not, performance will be better if you create Zotero translators for your popular sites first. WMF also has a fork of the Zotero translators which have slightly better coverage here.

Citoid is a nodejs app that also requires a working installation of Zotero's translation server, which uses the Zotero translators library, and xulrunner. Please note that the most recent version of xpcshell doesn't work with translation-server; the latest version known to work is 29.0.

Install nodejs and npm

Install nodejs 0.10 and npm. When you are using Ubuntu and depending on OS version you will not end up with the most recent version of nodejs.

sudo apt-get install nodejs npm
nodejs --version # should now print v0.10.x Note: not on Ubuntu Server 12.04 LTS, you end up with v0.6.x

For other systems, see:

• http://nodejs.org/download/
• https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager
• (Debian Squeeze, Wheezy) https://gist.github.com/x-Code-x/2562576

Install from scratch

Install and configure Zotero's translation server

See: Translation-server installation instructions

Note: The most recent version of xpcshell does not work with translation-server. Install version 29.0.

Install and configure citoid service

Get the code

If you want to do an anonymous checkout:

git clone https://gerrit.wikimedia.org/r/p/mediawiki/services/citoid

Or if you plan to hack citoid, then please follow the Gerrit 'getting started' docs and use an authenticated checkout url instead, such as:

git clone ssh://<user>@gerrit.wikimedia.org:29418/mediawiki/services/citoid

JS dependencies

Install the JS dependencies. Run this command in the citoid directory:

npm install

Modify config.yaml

Config.yaml contains the configuration for the citoid service. The defaults should work out of the box for development, however, they may need to be modified in a deployment set-up.

Run the server

You'll first need to run translation-server; see the directions on its github page, but generally from the translation-server directory you should run:

build/run_translation-server.sh

You should be able to start the citoid web service from the citoid directory using:

node server.js

This will start the citoid service on port 1970. To test it, navigate to http://localhost:1970 in your browser. You'll be able to test sample queries from this page.

Install Citoid extension

In order to have citoid functioning on your wiki in conjunction with VisualEditor, you'll need the following: VisualEditor and Parsoid, VisualEditor's Citation Tool, and the Citoid extension.

It is recommended that you have the following extensions in your extension folder: Extension:VisualEditor, Extension:Scribunto, Extension:Cite, Extension:TemplateData, and Extension:ParserFunctions, and Extension:Citoid.

VisualEditor and Citation Tool

1. Set up MediaWiki: Manual:Installation_guide
2. Set up Parsoid: Parsoid/Setup
3. Set up VisualEditor: Extension:VisualEditor
4. Set up Citation Tool: VisualEditor/Citation tool

Citoid extension

If you want to do an anonymous checkout:

git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Citoid

Or if you plan to hack citoid, then please follow the Gerrit 'getting started' docs and use an authenticated checkout url instead, such as:

git clone ssh://<user>@gerrit.wikimedia.org:29418/mediawiki/extensions/Citoid

Then add the following line to your wiki's LocalSettings.php after you have downloaded the extension:

wfLoadExtension( 'Citoid' );

Set the location to your citoid service instance in your wiki's LocalSettings.php

// If the wiki is being served over https, the https 
// protocol is required in the citoid service url; otherwise the 
// browser will block the request.
$wgCitoidServiceUrl = 'http://localhost:1970/api';

Configure Citoid on a Citoid-enabled wiki

The citoid extension must be configured using special TemplateData maps and a special citoid-specific message. It is currently deployed in all VisualEditor-enabled WMF-Wikis, but it must be configured before it can be used.

Configure special MediaWiki namespace Citoid message

Warning:

It is best if you add this message after each template has a template data map as specified in the following section. Otherwise, in the meantime empty templates will be inserted, because each template needs a template data map to be usable.

You'll need to configure a special MediaWiki: namespace message. This message maps the native citoid types (website, book, journalArticle) to the appropriate template (Cite web, Cite book, Cite journal). You should match a template to every single citoid type; there is no default behaviour if no template is matched to a particular type. It's better to have a bad match (there may be some fields in common between video liner notes and a book, or video liner notes and a video, for example) than none at all.

You may consider using en wiki's Template:Citation as a catch-all for types where there is no good type match as it is designed for this situation.

An sample namespace message is found here: Citoid/MediaWiki:Citoid-template-type-map.json

Every available citoid type is listed as a key in the sample namespace message.

Ensure each template specified in MediaWiki:Citoid-template-type-map.json has an 'citoid' maps value

Main article: Citoid/Maps TemplateData

Warning:

Fields issn and isbn require Arrays [] in the citoid map. All 'person' fields, e.g. author, editor, translator, contributor etc, require a 2D Arrays [[]] in the citoid map. See sample templateData below for examples.

Since Citoid has its own set of fields for each document type (for instance, the journal name is called 'publicationTitle' in citoid, but 'journal' in Template:Cite_journal), each Template must have TemplateData defined that creates a map between citoid's fields and the Template's field. Calling the map 'citoid' lets the citoid extension know which map to look for. If the map 'citoid' doesn't suit your purposes for use with say, a userscript, you may create a citoid service related map that is called something else; an unlimited number of maps with unique keys are allowed in the maps object. Note that you can only see TemplateData maps in edit mode; they are not visible in the TemplateData table.

Examples of maps objects that are compatible with the citoid extension are on beta:

• http://en.wikipedia.beta.wmflabs.org/wiki/Template:Cite_news/doc
• http://en.wikipedia.beta.wmflabs.org/wiki/Template:Cite_journal/doc
• http://en.wikipedia.beta.wmflabs.org/wiki/Template:Cite_web/doc
• http://en.wikipedia.beta.wmflabs.org/wiki/Template:Cite_book/doc
• http://en.wikipedia.beta.wmflabs.org/wiki/Template:Citation/doc

Troubleshooting VisualEditor Extension

Inspector does not appear in the toolbar

An icon for the inspector should appear in the toolbar menu. If the icon does not appear in the toolbar, it most likely means there's a problem with MediaWiki:Citoid-template-type-map.json. If there is no message at that location, or if the JSON is invalid, the inspector will not load. Alternatively, you may need to refresh your JavaScript cache.

Empty references appear

Empty references most commonly appear when the citation template being inserted contains no maps data, or if the maps data is there but not making it to the MediaWiki API. First, determine the template that the inspector is attempted to insert, for example, Template:Cite web/doc. View source of the template or documentation page to verify that

"maps": {

    "citoid": { 

is present and contains fields. Then verify that these data are making it to the MediaWiki API by visiting the API page, i.e. http://localhost/api.php?action=templatedata&titles=Template:Cite%20web/doc&format=jsonfm on your local installation, or https://en.wikipedia.org/w/api.php?action=templatedata&titles=Template:Cite%20web&format=jsonfm on en wiki.

If the maps object is present in TemplateData, but not in the API response, try editing the template where the TemplateData is transcluded i.e. Template:Cite_web (but making no changes) and saving it, a.k.a. a "null edit". There is a known bug with transcluded TemplateData where it can take a long for the API to update; null edits force the change.

If the response from the API looks okay, there may be an issue with the map itself.

The inspector is still "pending" after a really long time following insertion

This typically means there is a bug. If you open your JavaScript console, you will likely find error messages that will help you debug.

Access date is formatted differently on my wiki

The dates are in ISO format, which is an international standard. On the back-end, we're sticking to ISO and in the future all dates will all be in ISO, not just access date. This is because it is an unambiguous way to present the date in any language. If the community doesn't like the way this looks to the user, it is possible to edit the citation template to format the ISO dates to something that is standard in your language. For instance, you can add logic to the template such that if the date is detected to be in ISO yyyy/mm/dd format, the date is reformatted *to appear* to be dd/mm/yyyy on the page. However, if you do this, the underlying data (i.e. when you edit the wikitext, or the form in VisualEditor) will still remain the same.

Citoid has disappeared from my wiki

Please report this on Phabricator, and see that task for a temporary workaround.

Troubleshooting the citoid service

My favourite site isn't recognised by citoid and only gets basic information

The citoid service relies on the brilliant Zotero community for much of the "magic". We use Zotero translators to convert a page link into detailed information, and these translators need to be written for each site. You can see a list of all Zotero translators at https://github.com/zotero/translators – right now it is best for English language sources, and we need your help to improve coverage of other sites. Starting from one of the existing translators, and the general development framework instructions at https://www.zotero.org/support/dev/translators/framework, should help you write your own Zotero translator, which we would love you to contribute back to the wider Zotero community.

Testing for the translators using the "server" option or 'v' flag

To test with translation-server, download and install https://github.com/zotero/translation-server Your translator will need to have the 'v' flag enabled for 'browserSupport'. More here on that: https://www.zotero.org/support/dev/translators As an example, see https://github.com/zotero/translators/blob/master/3news.co.nz.js ; you will see there are a bunch of letters, one of which is v, which corresponds to translation-server. If server support is not enabled by testing it/then adding the 'v' flag, we won't be able to use the translator.

Hacking citoid

Installation

If you are using vagrant, you can enable the citoid role to hack on citoid.

Running tests

npm test only runs jshint.

mocha runs the full set of tests.

npm run-script coverage runs the tests and reports code coverage.

In order for all tests to pass, and you have manually installed Zotero, you must use our translator fork instead of the Zotero translators (if you are using vagrant and the citoid role this is already done for you):

git clone https://gerrit.wikimedia.org/r/mediawiki/services/zotero/translators

Citoid will also work with the official Zotero translators repo as well, but the output data will be different in some cases, and this will cause some tests to fail.

Another reason why some tests may fail erroneously is if your DNS will redirect invalid domains to a valid IP (Such as BT Internet's DNS); in some cases, this causes a 520 response instead of a 400 response to be returned. This can be fixed by configuring your internet connection to use OpenDNS or another DNS that does not do this.

See also

• Zotero translation server -- xpcshell-based Zotero translation server, and zotero-node, an abandoned nodejs equivalent
• Valid field names for each item type
• Zotero field mappings, including vs. some cite templates
• Citoid/Maps TemplateData
• en:Category:Citation_templates
• en:Help:Citation tools
• VisualEditor/Citation tool
• m:Research:Citoid support for Wikimedia references
• Translators testing
• http://citoid.wmflabs.org