Yes, we have a same problem on ckb wikipedia. It can be useful.

may be for fa.wikipedia or ckb.wikipedia we needs some normalization like

https://github.com/wikimedia/mediawiki-core/blob/master/languages/classes/LanguageAr.php

and https://github.com/wikimedia/mediawiki-core/blob/master/maintenance/language/generateNormalizerDataAr.php

Is this request about CirrusSearch or about LuceneSearch (deprecated)?

(In reply to Andre Klapper from comment #4)

Is this request about CirrusSearch or about LuceneSearch (deprecated)?

We need normalization for search box which is placed on the top pages.

In T72899#752615, @Aklapper wrote:

Is this request about CirrusSearch or about LuceneSearch (deprecated)?

I meant CirrusSearch

I'm having a hard time understanding the scope of this task. Could @TJones help? :-)

We need to specify the list of languages we are trying to do this for. The description mentions Persian, but alludes to Arabic, Urdu, Kurdish, Pushtu, and Uyghur, and the comments mention Sorani (ckb).

For each language, we'd need to figure out the exact mappings. (Persian is listed above, but I'd want to double check that the correspondences work in the other direction for each language—and there may be others glyphs that are relevant to other languages but not relevant to Persian, and so not listed here.) There may be more detail available in the github repos listed but I haven't looked closely.

Then we have to figure out which language analyzers are being used for each language. Arabic, Persian, and Sorani have their own analyzers. The fallbacks (T147959), though very imperfect, are the status quo, so we'd have to see what's going on there.

For each analyzer being used (possibly including the default), we'd need to unpack the built-in ES analyzer so we can modify it. Doing this for French and others has given unexpected results—generally not bad, mostly improvements, with the few regressions being readily fixable. Figuring all that out requires testing per language, and I'd really want to be careful the first time we did it to an Arabic-script analyzer.

After the unpacking, actually setting up the mapping is very little work.

Once that's done, the wikis in question need to be re-indexed. Arabic Wikipedia has already been done for BM25, and others may happen before we work on this, in which case the change going live could take a while—until the next time we re-index. (Though that's something we need to get better at being able to do, and doing it for the projects in a handful of languages is less effort than doing it for almost everything, as we are with BM25.)

Does that help?

Change #1024764 had a related patch set uploaded (by Tjones; author: Tjones):

[mediawiki/extensions/CirrusSearch@master] Normalize Arabic variants of kaf, yeh, heh

https://gerrit.wikimedia.org/r/1024764

I looked at all of these as best I could, and I decided on a general mapping to standard Arabic forms internally. Arabic does that to some degree, as does Persian! And for the languages without custom stemmers and stop word filters, the character used internally doesn't matter, as long as the desired words can find each other.

Things were complicated by the special characters for isolated, initial, medial, and final forms that many Arabic letters have, but I got it under control in the end!

Sorani/Central Kurdish (ckb) didn't need any normalization. It already normalizes to the Persian forms (though Persian often doesn't!) and didn't have any benefit from the custom normalization I tried to brew up for it—plus a lot of things broke!

More details in my full write up on MediaWiki.

Change #1024764 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Normalize Arabic variants of kaf, yeh, heh

https://gerrit.wikimedia.org/r/1024764