Better identification of

• which email gives the transbot problems,
• what and where in the email gives the transbot problems

in its error response emails.

When the translation tool has issues with the email, an email is sent from transbot-no-reply@icann.org with
the static subject line "Unable to translate your email to ICANN lists" and a template text like

Dear <sender>

Thank you for your participation in the ICANN email list new-transbot-en.
You are getting this email because we were unable to translate your post automatically.
It violated one or more of the formatting rules we must impose to make translation possible.

A complete description of the formatting rules is available at:
https://community.icann.org/x/aYtEAg

In preparing your post for translation, we found the following format violations in your message:
    <issue, usually Sentence punctuation must be followed by a space>

Please edit your post and send it again.
Thank you.

Problem is there is no identification of which email this is referring to - perhaps the subject line of the problem email should be included either in the subject line or in the body of the email.

The format violations message doesn't say WHERE the error is in the email. If its a long email, then how can users identify and correct such issues. Persons getting these messages and not easily seeing where the problem is aren't likely to understand how to do future emails better and the warning messages becomes more of an hindrance.

Perhaps a workaround is to have the text of the email in the error email and identify what section the transbot has issues with.

Other questions :

• Why is the test required? What is this test trying to solve?
• How are domain names handled? Since domain names can't have spaces, do domain names without beginning with http:// trigger the error?
  
  

 Tool now identifies the subject line of the problem email in the body of the email

Missing emails from one list to another

In 2018, it was noticed that many emails are being silently dropped from one list, mainly emails from lacralo-es not being sent to lacralo-en

Comparing :
lacralo-en archives at https://atlarge-lists.icann.org/pipermail/lac-discuss-en/ and
lacralo-es archive at https://atlarge-lists.icann.org/pipermail/lac-discuss-es/

Why is this happening?

<DNT> tag isn't case sensitive.

Using <DNT> tag works to not translate text, but <dnt> does not.
The tool should be able to treat the DNT tag, regardless of case.

The <DNT> tag will be seen in the original email but not in the translated one.

See EN and ES messages.

Q: Should the <DNT> tags be removed in the original email? I'm thinking it should

To achieve this, it would be necessary to re-architecture the transbot code.

Upon investigating the issue it was discovered this is a known limitation, rather than a bug in the existing code.

The request has been recorded as something to consider in a future iteration of the translation tool.

Re: attachments, new-transbot lists have a message and attachment limit of 200K

Given many PDFs will be larger, will be hard to test unless message size limit is raised

Tested with a smaller attachment, the attachment does go through. See EN and ES

The new-transbot list email size limit has been increased to 400K.

This is enforced for the entire email, including text and attachment.

Handling an email sent to both new-transbot-en and new-transbot-es lists at the same time.

When such an email to both lists happens, some emails don't get translated.

A decimal number in an email e.g 3.9MB will trigger the error message

"Sentence punctuation must be followed by a space"
See EN email

Re: attachments, new-transbot lists have a message and attachment limit of 200K

Given many PDFs will be larger, will be hard to test unless message size limit is raised

Tested with a smaller attachment, the attachment does go through. See EN and ES

The new-transbot list email size limit has been increased to 400K.

This is enforced for the entire email, including text and attachment.

Subject line in body of translated email has the sender and first line of the email on the same line, when the sender and first line should be on separate lines. Two examples:

• EN (original): http://mm.icann.org/pipermail/new-transbot-en/2016-February/000080.html ES (translated): http://mm.icann.org/pipermail/new-transbot-es/2016-February/000068.html
• EN (original) : http://mm.icann.org/pipermail/new-transbot-en/2016-February/000083.html ES (translated): http://mm.icann.org/pipermail/new-transbot-es/2016-February/000071.html

The converted body text was included immediately after the subject and from line in translated emails. A new line character was inserted between sender name and first body text line to separate them.

Initial testing complete; additional testing in progress.

Example: https://community.icann.org/x/AofDAw

There is an empty space in the beginning of most (but not all) lines in the translated email

• EN (original): http://mm.icann.org/pipermail/new-transbot-en/2016-February/000080.html ES (translated): http://mm.icann.org/pipermail/new-transbot-es/2016-February/000068.html

After investigation it appears empty spaces may have been added by the email client or by Google Translate API.

Space removed before all translated lines in mailing list emails to resolve the issue.

Tested with mail IDs Outlook, Yahoo, Gmail. Verified that translated emails are not indented; empty spaces are not appearing at the beginning of lines.

Example: https://community.icann.org/x/vYPDAw

At the end of the message, the sender's name starts with a lower case ('dev Anand', although it is 'Dev Anand' in the original message
EN (original): http://mm.icann.org/pipermail/new-transbot-en/2016-February/000080.html ES (translated): http://mm.icann.org/pipermail/new-transbot-es/2016-February/000068.html

Google Translate understands 'Dev' as an abbreviation and as a rule converts it to all lowercase.

Name was hardcoded to fix 'Dev' to begin with a capital letter.

Tested with mail IDs Outlook, Yahoo, Gmail. Verified that name is appearing correctly with first letter capitalized.

Example: https://community.icann.org/x/ooPDAw

'Transbot' is mis-spelled as 'tansbot' (third line from the bottom)
EN (original): http://mm.icann.org/pipermail/new-transbot-en/2016-February/000080.html ES (translated): http://mm.icann.org/pipermail/new-transbot-es/2016-February/000068.html

Misspelling was hardcoded. Applied hardcode fix to correct spelling of 'tansbot' to 'transbot' in translated emails.

Tested with mail IDs Outlook, Yahoo, Gmail. Verified that spelling is now appearing correctly as 'transbot.'

Example: https://community.icann.org/x/oILDAw

the transbot can't handle cedilla - as At-Large Staff signature lists a staff member with a cedilla in her name, any message from At-Large Staff will result in a message not translated
See

• Email with cedilla  EN: http://mm.icann.org/pipermail/new-transbot-en/2016-April/000085.html  ; ES : http://mm.icann.org/pipermail/new-transbot-es/2016-April/000073.html
• Email with cedilla removed : EN : http://mm.icann.org/pipermail/new-transbot-en/2016-April/000094.html ; ES : http://mm.icann.org/pipermail/new-transbot-es/2016-April/000082.html

The April thread with the subject line "CALL FOR MEMBERS: At-Large Public Interest Working Group" on EN : http://mm.icann.org/pipermail/new-transbot-en/2016-April/thread.html and ES : http://mm.icann.org/pipermail/new-transbot-es/2016-April/thread.html showed how the issue was isolated after several variations of the original email were tried.

This is a critical bug, as any cedilla in any word in an email would result in the email not being translated.

The issue has been resolved specific to the reported cases of broken emails caused by the cedilla character in the AL Staff signature.

As a larger issue, it is still in progress. Efforts around the reported case led to wider investigation into how the transbot and email applications handle Unicode characters. This is important UTF-8 compliance work and requires extensive testing.

Recent tests with a wider set of characters using Outlook have been successful. Tests with those same characters have been inconsistent with Gmail and Yahoo. The team is continuing to research, test, and make progress.

Examples: https://community.icann.org/x/pYrDAw

The phrase "This Working Group is open to interested members of the At-Large community." gets translated to

"Este grupo de trabajo está abierto a los miembros interesados \u200b\u200bde la comunidad de alcance."

Not sure why it repeatedly happens for that phrase: See
EN: http://mm.icann.org/pipermail/new-transbot-en/2016-April/000090.html
ES: http://mm.icann.org/pipermail/new-transbot-es/2016-April/000078.html

The issue was related to zero-width space, which was being injected by Google Translate API.

Zero-width space is used after characters that aren't followed by a visible space, but after which there may be a line break (source). It was encoded into the Unicode and was appearing as /u200b.

To fix, it was replaced in the translated text with no space.

Tested with mail IDs Outlook, Yahoo, Gmail. Verified that phrase is translated correctly without additional characters.

Example: https://community.icann.org/x/t4PDAw