How to Prepare an Alignment Project for Publishing

jag3773 · April 24, 2019, 6:45pm

So, you’ve finished aligning your translation to the UGNT using translationCore and now you are ready to publish it so that others can benefit from your work. Here’s what you’ll need to do…

The short version looks like this: upload to DCS, identify/create project to merge into, rename USFM3 alignment file, modify manifest.yaml file.

Upload to Door43

First, if you haven’t already, upload your project from tC to Door43 (Content Service) (DCS). From the project’s screen, click the three dot menu on the project you’re working with and select “Upload to Door43”.

You’ll then get an alert box telling you where the project was uploaded to, click on the link.

Inside your uploaded project you’ll see 2 USFM files, one of them is a special USFM3 formatted file that contains the alignment data from your project. This special one will be the one with the longer name, the highlighted one in the image below.

Merge

You’ll now want to either create a new project on Door43 or identify an existing project that you want to merge this book into. For example, let’s say we are working on adding or updating the alignment data for the Indian Revised Version - Hindi, in this case we’ll want to be updating the project that is published here: Door43-Catalog/hi_irv - hi_irv - Door43 Content Service.

The first step would be to fork that project, now you have a copy of repository that you can make changes to. Once you’ve done that you can now download the new USFM3 alignment file from your tC project repository and upload it into your fork of the hi_irv repository.

Continuing this example, we could start by downloading hi_irv_jas_book.usfm from https://git.door43.org/shojojohn/hi_irv_jas_book . I like to do this by using wget on the command line, like so: wget https://git.door43.org/shojojohn/hi_irv_jas_book/src/branch/master/hi_irv_jas_book.usfm.

Next, you’ll need to rename the file to match our USFM file naming convention, in this case rename hi_irv_jas_book.usfm to 60-JAS.usfm. It’s also good at this point to make sure the headers in the USFM3 file are correct and you’ll probably need to add \usfm 3.0 after the \id line too.

Now you can add this file to your fork of the Door43-Catalog/hi_irv project. Do this by using the Upload file button you see in DCS.
30%20PM

The remaining step is to modify the manifest.yaml file as appropriate to indicate the changes you are making. In this case, we’ll need to add an entry to the projects array like so:

  -
    title: 'याकूब '
    versification: 'ufw'
    identifier: 'jas'
    sort: 59
    path: './60-JAS.usfm'
    categories: [ 'bible-nt' ]

You will also need to adjust the version number, issued date, and modified date. The subject and `format fields should be set like so:

  format: 'text/usfm3'
  subject: 'Aligned Bible'

Pull Request

At this point you can issue a pull request against the upstream project, in our example that would be Door43-Catalog/hi_irv - hi_irv - Door43 Content Service.

jag3773 · May 28, 2019, 8:18pm

Note that I wrote some scripts to help simplify this process and make it (hopefully) less error prone.

Get Alignment Data

Downloading the alignment data from a tC repo is not really straightforward (as seen above), especially figuring out which file is the right one and then naming it properly locally.

I wrote this get_aligned_usfm.sh script to streamline the process. Essentially, you pass in a repository link and it will identify the correct file to download and it will name it according our RC Bundle spec for USFM files (e.g. Titus is 57-TIT.usfm).

If you put it in your bin directory you can use it like this:

~/bin/get_aligned_book.sh https://git.door43.org/pjoakes/en_ust_1ti_book

As the comment says, make sure you also have the get_bible_book.py script available so that it can find the right book name to use.

github.com

unfoldingWord-dev/tools/blob/develop/usfm/get_aligned_usfm.sh

#!/bin/bash
#
# Expects argument of repo path (like https://git.door43.org/shojojohn/bn_irv_1th_book)
# Convert that path to filename for download (like https://git.door43.org/shojojohn/bn_irv_1th_book/raw/branch/master/bn_irv_1th_book.usfm)
# Downloads and properly names file in cwd
#
# Must have get_bible_book.py in ~/bin/ directory

TCBOOKNAME="${1##*/}"
GETURL="$1/raw/branch/master/$TCBOOKNAME.usfm"
CODE=`echo $1 | cut -d '_' -f 3`
USFMBOOK=`python ~/bin/get_bible_book.py $CODE`

wget "$GETURL" -O "$USFMBOOK"

Manifest Update

Another part of this is simply updating the manifest file. This script does a few things:

Verifies that every USFM file in the current working directly is also represented in the manifest (or yells at you)
Verifies that every USFM file listed in the manifest is also present in your current working directly (or yells at you)
Checks for the zaln marker in the USFM files, if so it flags it as a USFM 3 project and updates the USFM files and the manifest accordingly
Updates the issued and modified fields to today’s date

You just run this when in the project’s directory and it should do the right thing.

github.com

unfoldingWord-dev/tools/blob/develop/usfm/manifest_update.sh

#!/usr/bin/env bash
#
# Simple script to update the manifest.yaml file with a few things
# Should be safe to run multiple times on same project
# Should be safe to run on any type of project

# If running on Mac, we need gsed
SED=`which sed`
if [[ -f /usr/bin/sw_vers ]]; then
    if [[ ! -f /usr/local/bin/gsed ]]; then
        echo "Please install gsed"
        exit 1
    fi
    SED="/usr/local/bin/gsed"
fi

# Verifies that the files listed in the manifest are present in the cwd
echo "Ensure every file in manifest is present..."
for x in `grep -ow '[0-6][0-9].*.usfm' manifest.yaml`; do
    [[ -f $x ]] || echo "--> MISSING FILE: $x"

This file has been truncated. show original

Notice

Make sure you are using git on the repositories/projects you are working with, or at least some way to easily revert changes. The above scripts do make changes and aren’t guaranteed to do the right thing – I’ve only tested them on my mac.