Will the real rules please stand up?

 One of my pandemic projects is to map the differences between a pointed and an unpointed text of the Hebrew Bible. I am doing this using the very memorable and capable left to right version of the square text called SimHebrew. 

I had 'my own version' of a left-to right abbreviation  in capital letters and with a few punctuation marks for the gutturals aleph and ayin. But I converted to the lower case version developed by Jonathan Orr-Stav. (Such a Latin-letter code is an abbreviation because it takes one byte as opposed to 7 bytes in rendered Unicode, and one byte as opposed to 2 internally. Using a Unicode database is really awkward for me and the technology was new 10 years ago so it was a non-starter.) 

My method of data capture is to use the unpointed Mechon-Mamre text that can be downloaded from their site (one book at a time) and to run it through the SimHebrew converter here. Then to manipulate the text in notepad until I have legitimate insert statements for my database. This is somewhat prone to error but eventually I get a clean script. (I use a combination of Word and Notepad for the necessary global changes.)

After this I create a temporary table that matches: book, chapter, a conversion of the alef-betic verse number to a real verse number, and a word by word Hebrew word in SimHebrew, full pointed text word from the Leningrad codex, the stem code, raw word form, and semantic domain from my database, the word number relative to the start of the verse, and the word id (assigned by an Oracle sequence and connecting to my word table).

I recently added 2 Kings to my data. At first I assigned the wrong chapter numbers - a little oversight. That resulted in over 5000 differences in my calculations. When I fixed the issue, the differences went down to about 230. I then discovered that M-M is treating דִּבְיוֹנִ֖ים (dbivn) which they render as dung of a dove as two words (db ivnh - apparently a poor substitute for salt). Fixing those 4 instances dropped my mismatches by 100. Then word by word I work through the remainder until the differences are accounted for.

My program that calculates the words also suggests a code snippet that will fix any discrepancy (as long as I put it in the right place in the code.) The program as it stands gets 99% of the words right on first pass now. So my predicted SimHebrew Bible is about 99% right. Sorry - for the Bible, that's not good enough.

About this time in the history of the project, I ask, Is there a better way? I know I can finish the way I have begun and my brute force spelling changes are next to 0. I have over a third of the words in my test data now and over 75% of the stems represented. But the code is specific to prefix and suffix and sometimes to particular vowel combinations in the WLC. What are the real rules? 

Native speakers who 'just know' the pronunciation - what are they really doing? Certainly they have retention in memory by word form, by context, and by stem. But can I get the program to discover the shortcuts that people use? And in some ways see what they are doing. (And thereby discover the nature of the evolution of language usage.)

Here's an example: the 31 uses so far of עצר (yxr)

For yxr, the following rules are noted. 

• First you see that the holem is rendered as /o/ in lines 1 and 2 and others. This is default, but some words with some exceptions do not render the o. 
• Next (line 3) you will note that the qamats is not rendered as /o/ (or vav for rtl Hebrew) but it is in line 4, but the qamats under the second letter of the stem is rendered with prefix /i/ and /t/ in each case without suffix. 
• Tsere is rendered as /i/ in line 6 and 12 - but only the tsere under the last letter of the prefix. (I would like to color code these but the new editor absolutely ruins rtl display with embedded color coding.) 
• Notice that the final h is dropped in line 19. 
• Lines 26 and 27 would have rendered the qamats in the closed syllable as /o/ if the suffix had not been /u/. That's a general rule - but there are exceptions here too and exceptions to the exceptions.

Ref (book: chap: vs: (word within vs)	Stem	Word Form	Morph	Sim Source	Sim Calc.	WLC Word	Domain	Rendering
2 Chronicles 14:10(30)	yxr	iyxr	i/yxr	-iyxvr	-iyxor	־יַעְצֹר	BOUND	יעצר will coerce
1 Chronicles 29:14(7)	yxr	nyxr	n/yxr	-nyxvr	-nyxor	־נַעְצֹר	BOUND	נעצר contained of
2 Chronicles 13:20(2)	yxr	yxr	yxr	-yxr	-yxr	־עָצַר	BOUND	עצר did coerce
2 Kings 4:24(9)	yxr	tyxr	t/yxr	-tyxvr	-tyxor	־תַּעֲצָר	BOUND	תעצר do detain
2 Chronicles 7:13(2)	yxr (5)	ayxr	a/yxr	ayxvr	ayxor	אֶעֱצֹר	BOUND	אעצר I contain
1 Kings 8:35(1)	yxr	bhyxr	bh/yxr	bhiyxr	bhiyxr	בְּהֵעָצֵר	BOUND	בהעצר when is contained
2 Chronicles 6:26(1)	yxr	bhyxr	bh/yxr	bhiyxr	bhiyxr	בְּהֵעָצֵר	BOUND	בהעצר when are contained
Amos 5:21(6)	yxr	byxrticm	b/yxr\ticm	byxrvticm	byxroticm	בְּעַצְּרֹתֵיכֶם	COVENANT	בעצרתיכם in your conclaves
2 Kings 9:8(9)	yxr	vyxvr	v/yxvr	vyxvr	vyxur	וְעָצוּר	BOUND	ועצור the contained
1 Kings 21:21(11)	yxr (10)	vyxvr	v/yxvr	vyxvr	vyxur	וְעָצוּר	BOUND	ועצור the contained
Proverbs 30:16(2)	yxr	vyxr	v/yxr	vyvxr	vyoxr	וְעֹצֶר	BOUND	ועצר and contained
1 Chronicles 21:22(17)	yxr	vtyxr	vt/yxr	vtiyxr	vtiyxr	וְתֵעָצַר	BOUND	ותעצר that may be contained
2 Kings 17:4(20)	yxr	viyxrhv	vi/yxr\hv	viyxrhv	viyxrhu	וַיַּעַצְרֵהוּ	BOUND	ויעצרהו so detained him
Job 4:2(5)	yxr	vyxr	v/yxr	vyxvr	vyxor	וַעְצֹר	BOUND	ועצר but contain
Psalms 106:30(4)	yxr (15)	vtyxr	vt/yxr	vtiyxr	vtiyxr	וַתֵּעָצַר	BOUND	ותעצר and was contained
Job 12:15(2)	yxr	iyxr	i/yxr	iyxvr	iyxor	יַעְצֹר	BOUND	יעצר he contains
2 Chronicles 2:5(2)	yxr	iyxr	i/yxr	iyxvr	iyxor	יַעֲצָר	BOUND	יעצר contains
1 Kings 18:44(19)	yxr	iyxrch	i/yxr\ch	iyxvrç	iyxorc	יַעַצָרְכָה	BOUND	יעצרכה detain you
2 Chronicles 22:9(27)	yxr	lyxr	l/yxr	lyxvr	lyxor	לַעְצֹר	BOUND	לעצר to coerce
Psalms 107:39(3)	yxr (20)	myxr	m/yxr	myvxr	myoxr	מֵעֹצֶר	BOUND	מעצר through coercion of
Proverbs 25:28(8)	yxr	myxr	m/yxr	myxr	myxr	מַעְצָר	BOUND	מעצר containment
2 Chronicles 7:9(4)	yxr	yxrt	yxr\t	yxrt	yxrt	עֲצָרֶת	COVENANT	עצרת a conclave
Joel 1:14(4)	yxr	yxrh	yxr\h	yxrh	yxrh	עֲצָרָה	COVENANT	עצרה a conclave
Joel 2:15(7)	yxr	yxrh	yxr\h	yxrh	yxrh	עֲצָרָה	COVENANT	עצרה a conclave
2 Kings 10:20(4)	yxr (25)	yxrh	yxr\h	yxrh	yxrh	עֲצָרָה	COVENANT	עצרה a conclave
Job 29:9(2)	yxr	yxrv	yxr\v	yxrv	yxru	עָצְרוּ	BOUND	עצרו contained
2 Chronicles 20:37(19)	yxr	yxrv	yxr\v	yxrv	yxru	עָצְרוּ	BOUND	עצרו could be coerced
2 Kings 14:26(10)	yxr	yxvr	yxvr	yxvr	yxur	עָצוּר	BOUND	עצור coercion
Jeremiah 36:5(7)	yxr	yxvr	yxvr	yxur	yxur	עָצוּר	BOUND	עצור am detained
1 Chronicles 12:1(7)	yxr (30)	yxvr	yxvr	yxvr	yxur	עָצוּר	BOUND	עצור he contained himself
1 Kings 14:10(12)	yxr	yxvr	yxvr	yxvr	yxur	עָצוּר	BOUND	עצור those who are contained

All that is pretty straightforward. But what are the real rules? 

Some of my rules are miles long conditions; strings of stems with prefix and suffix combinations and the occasional appeal to an odd Unicode value. 

The 'rules' are long for when to render hireq as /i/.

• 200 distinct stems which do not contain a yod are allowed to render hireq as yod without exception.
• Another 281 allow hireq as yod on an exception basis.
• Only 7 stems containing a yod disallow hireq as yod.
• Another 46 disallow it on an exception basis.
• Hireq is usually ignored in a closed syllable - but there are exceptions and exceptions by word form to the exceptions (only 8 stems).
• Hireq is rendered as /ii/ for several reasons. When the prefix is i, and for tsere, patah, and qamats occasionally. What! these also are rendered as o or v sometimes. Who can know?

Any ideas?

I have been debating whether to design a data table for the rule combinations. I don't want to do it unless I cannot simplify the code. Last time I asked I discovered several simplifications. Looking for more...

Here's the rule for qamats becoming vav (RTL) or /o/ (SIM): I have converted the code to 'English'

Unconditionally for stems ahl, anih, arc, avn, bvw, ctl, grn, iq+n, iqwn, irqym, nvh, rnn, yziali, zvh, 

(+ is my single letter internal abbreviation for tet ט.)

or the stem is +rk and prefix suffix combination in (none, cm) 

or the stem is +hr and prefix suffix combination in (none, t, c, t)

or the stem is abd and prefix suffix combination in (b, n)

or the stem is acl and prefix suffix combination in (l, h, b, nu) 

or the stem is adm and v_prefix = m and the first part of the word is m with a schwa

or the stem is amn and the word form is amn and there is no tsere under the second stem letter

or prefix suffix combination = b, h

or the stem is ark and v_suffix in (ti, vt)

or the stem is bit and v_suffix in (icm)

or the stem is bzz and prefix suffix combination in (i, vm)

or the stem is cvl and prefix suffix combination in (v, clu)

or the stem is csh and prefix suffix combination in (none, u)

or the stem is dbr and prefix suffix combination in (b, c) 

or the stem is gal and prefix suffix combination in (l, c)

or the stem is gbh and prefix suffix combination = l, h

or the stem is gml and prefix suffix combination in (none, h,none, c) 

or the stem is hrh and prefix suffix combination in (vh, tihm)

or the stem is ivn and prefix suffix combination in (none, none)

or the stem is iwr and prefix suffix combination in (none, o, b, o)

or the stem is imn and prefix suffix combination in (m, none)

or the stem is isd and prefix suffix combination in (b, i)

or the stem is knn and not (suffix in (u) and there's a qamats under the second letter) and the domain is not in (PERSON, LOCATION)

or the stem is kq and v_suffix not in (u)

or the stem is kgg and prefix suffix combination in (none, i) 

or the stem is krb and prefix suffix combination in (none, h, l, h) 

or the stem is krm and prefix suffix combination in (none, h, vb, h) 

or the stem is kwc and prefix suffix combination in (none, i) 

or the stem is kzq and prefix suffix combination in (b, h, b, nu) 

or the stem is lcd and prefix suffix combination in (l, h) 

or the stem is mlc and prefix suffix combination in (l, o, b, o, c, o) 

or the stem is mvt and prefix suffix combination in (h, h) 

or the stem is pyl and prefix suffix combination in (c, h) 

or the stem is q+n and prefix suffix combination in (none, i)

or the stem is qdw and v_suffix in (im, iv)

or the stem is qra and prefix suffix combination in (none, nu, none, ic, none, im) 

or the stem is qrk and prefix suffix combination is none, h

or the stem is rb  and prefix suffix combination in (b, none) 

or the stem is sll and prefix suffix combination in (none, vh) 

or the stem is rbb and prefix suffix combination in (v, none)

or the stem is rbb and prefix suffix combination in (hb, none)

or the stem is rkb and prefix suffix combination not in (none, none)

or the stem is rkx and prefix suffix combination in (l, h)

or the stem is tmm and prefix suffix combination in (b, none) and the first two letters of the word are not m

or the stem is wby and prefix suffix combination in (m, h, l, h) 

or the stem is wcb and prefix suffix combination in (b, o) 

or the stem is wcr and prefix suffix combination is l, h

or the stem is wdd and prefix suffix combination in (i, m)

or the stem is wmr and prefix suffix combination in (a, none,vl, h,none, h)

or the stem is wp+ and prefix suffix combination in (none, h) 

or the stem is wr and suffix in (c, rc)

or the stem is wrw and prefix suffix combination in (none, ih) 

or the stem is xrp and prefix suffix combination in (none, h) 

or the stem is ybd and prefix suffix combination in (l, h)

or the stem is yni and prefix suffix combination in (none, nu) 

or the stem is yzz and prefix suffix combination in (none, i) 

or the stem is zcr and prefix suffix combination in (b, nu) 

or the stem is yzr and prefix suffix combination in (l, ni, l, u,none, ni,none, nu)

then

the qamats under the first letter of the stem becomes /o/.

And by the way if the stem is qdw and v_suffix in (im) then the schwa under the first letter becomes /o/!

(or maybe only v)