A process for maintenance of the music once it is extracted from the database

At some point I hope to have EPUB's for every chapter of the Bible, one verse at a time with the music. I designed what is likely to be my final product to help people learn the music of the Hebrew Bible. 

Here is a quick look at Isaiah 40 verse 1.

The music embedded in the text of Isaiah 40, verse 1

And here's the text:

Isaiah 40: (Verses 1 to 1) Syllables: 14; Longest recitation: 4; Tenor: g 42.86%;
Ornament density: 0%; Average phrase length: 7.

1 Comfort, comfort, my people, says your God. (1-4-1)
א נחמ֥ו נחמ֖ו עמ֑י יאמ֖ר אלהיכֽם	8 6		a nkmu nkmu ymi iamr alohicm

Notice how the music moves based on the accents under the text. Count them (5) and observe the five movements from the starting note to the mid point and back.

Everything is in the database. I am just beginning to work on this chapter and it fits my purpose in this blog post. It took a few short minutes for me to produce that music and text directly from the database. I didn't type or scribe any of it. I had to use a png image instead of svg on this platform since blogger doesn't (yet) support svg. 

We wrote data-driven programs. If the data was well-designed, it was easy to get the required output from it.

Over the last few months, I have been getting the data out of the database and into epubs where normal people could maintain it if necessary. It is becoming simpler.

For those sections that I have finished, the music and the data surrounding it can be maintained by anyone who can use Musescore and Calibre. No other dependencies needed - except to be able to rename files. No database knowledge required.

Finally I figured out a way to generate the html and .bat files so that they have the best chance of corresponding to individual lines of music. Roughly 30 or 31 syllables will fit on a page width depending on the ornament density of the verse. Most verses fit within three lines of music on an iPhone, making it possible to have the Bible in Hebrew and English with its embedded music in your pocket.

Music Files

I will deliver all the necessary music files and verified batch rename files to my publisher if we have success, so that if people wanted, for instance, the base music for a chapter, it would be available in an mscz (music zip) file. Load the mscz file, change the page length to the length you want, remove the line feeds as desired, and you have the full score for a chapter. Arrange and perform as you wish. The music itself is not copyright. It is part of the Bible and derived by a key that is discoverable from the way the te'amim are used in the Hebrew text.

Possibly some day, Calibre or some other program or avatar will be able to sing the music for the reader, but that was not possible the last time I checked. And I won't be checking any time soon if I am going to get all the data out and ready for study. I'm sorry that more people don't follow Wm Byrd's injunction to learn to sing. And by the way, learn to read music also.

I have given away the music xml and pdfs many years ago, (see the music page) but I had not created and sculpted the music images into scalar vector graphics (svg) form, one verse at a time. Nor had I fixed the WLC data to conform to Aleppo as much as I have this time round. It is still not perfect but it is closer to the original, and I am not convinced that even Aleppo is clear of the kinds of error I have encountered.

Maintaining the data is trickier than just using the music to prepare an arrangement. The music images are directly dependent on the Hebrew text. The four versions of the Hebrew text seen above in the image and text should remain consistent if the epub needs to be updated.

This is my process for getting the data out of the database into the e-pub:

1. Fix premature descents to the tonic – compare WLC verse by verse with MG Crown Aleppo codex. Check all internal descents to the tonic against mgketer site. I do this before generating the music. This step can be done for multiple chapters -- I keep going until I get tired. (Code to compare the two versions would be difficult to write. WLC uses a different level of Unicode from mgketer, coding sequences of diacritics differ even within the WLC, and there are a host of irrelevant notes and other comments that need to be ignored.) So far in about 300 chapters and about 6500 verses, I have corrected about 700 errors in the WLC. That's just over a 10% error rate. In what -- just over 100 years of copying? (Copyists get tired too!) The database remembers my changes so I can tell what I have done and when. I use three windows: 
• my proprietary update screen for the database, 
• the mgketer chapter, (in Hebrew only).
• a filter on my shortcut work file showing the notes to isolate the verses I suspect are in error. 
2. Generate music XML using my music generation page. This can be done for multiple chapters that have been verified. Takes a few seconds per chapter.
3. Run the data and the batch file for the chapter to html format. This program is one of many I have written for extracting data in various formats.
4. Open the music with Musescore, load the style file for one line per page. Open the bat file with a text editor and check the music 'pages' against the batch file. Page size is in the image:

1. Load the one-line style. [Saved in my Google Drive here]
2. Mark all beams as disconnected -- they will disappear (use multi-select). Make all triplets invisible – note stems are already invisible from program that generates xml.
3. Force returns on all verses – one verse per 'page'. (Every verse ends with a rest and a barline.) Fix alignment of text and check for slurs that displace text or make the lyrics line too low, adjust to under or over as needed.
4. (Poetry only -- add breath for ole-veyored – should automate this but tricky).
5. Adjust margins if needed.
6. Save to mscz file.
7. Verify the batch file against the music. Must agree on pages. Possible to insert a blank page and delete it later -- mark the bar as not included.
8. Export svg to work area.
9. Use saved .bat file in cmd window or equivalent to rename the files.
10. Load svg files and html into e-pub.
11. Verify. (Calibre verify function will find errors in renaming if you miss them).
12. Delete the svg files in the work area.

This process takes from 10 minutes to an hour for longer prose chapters.

All Musescore files and batch rename files are saved in case steps 1 to 12 need repeating. Spot corrections are possible but tricky since the text and music are so integrated. E.g. a change in a word affects the music, the staff text, the Hebrew words with te’amim also and potentially, bar numbering. 

If a change is required, load the mscz file, make the changes and repeat the process from step 6.

I've documented this for my use -- but if anyone takes over what I am doing, who knows, it might give them some ideas.

What would motivate someone to do this task when nearly four-score years? Well, it's fascinating and it changes how you read. I hope that we (humanity) might start to read the Biblical text with love rather than our own fear and petty prejudices. So that we might learn to comfort each other, the people that Isaiah is referring to -- indirectly of course.