Skip to main content

XPages and RichText (part I) - ACF

 This post is long overdue. I originally planned to write it to document the issues and solutions while working on the app, but as the problems kept appearing I wanted to make sure that I have all the answers. Now I know that I probably won't have them in the near future as I'm seeing new issues even after having the app in production for some time. 

The project was about a webization of an existing app that served well in Notes client UI for many years. It's a classic support/helpdesk app that once used to be a great example of Notes flexibility and integration of mail functions with app code. After all the mail database was just another app using the same APIs (mostly). Unfortunately, this is not the case for XPages as iNotes or Verse has nothing in common with XPages, so the APIs are not battle-tested for all the scenarios. I remember doing a similar project during the early days of XPages and I must admit that it was more painful, but mostly due to a lack of community resources and experience. Some of the issues that I'll be mentioning are fixed in the upcoming 12.0.2 version, so it may get easier to maintain the app eventually. 

There are still many great blog posts about XPages, RichText and MIME issues, so I'll try to refer to them if possible. Unfortunately, some of the old blogs are no longer available, which means that links e.g. from StackOverflow answers are no longer helpful. 

Before starting the project we did a proof-of-concept to see if it was even possible to implement all the mail-related functions in a way that will work for the users. The app uses all classic mail functions like reply to all with history (and without attachment) and needs to be backward compatible, so the code must work for existing RichText items and new data stored in MIME. The PoC was successful, so we started the implementation. 

I won't be mentioning the issues in the order as we have discovered them, but rather based on their impact on the final (currently) solution.

Problem 1 - To ACF or not to ACF

Active-content-filtering is a really important feature for security, especially when we are working with emails from the Internet. We had ACF enabled (in the default) settings during the development and testing of the application. It worked well. Unfortunately, once we deployed the app to production, we started to see issues with some received emails.

ACF crashing on semi-invalid messages

The first problem that we encountered was caused by invalid inline image data in emails. The details are not that important, simply the image claimed to have inline base-64 value, but there was none. Something like:

<img src=3D"data:image/png;base64,cid=
:116c8444-683e-423d-964a-0b914d8e4b01" style=3D"position: static !importa=
nt;" title=3D"Call: " />

 The result was ACF completely crashing in a way that the page was not loaded:

Exception
Error while executing active content filter
Bad character exits in Base64 array
Array index out of range: -1

com.ibm.xsp.FacesExceptionEx: Error while executing active content filter
com.ibm.xsp.acf.ACFProcessor._processMarkup(ACFProcessor.java:106)
com.ibm.xsp.acf.ACFProcessor.processMarkup(ACFProcessor.java:92)
com.ibm.xsp.context.FacesContextExImpl.filterHtml(FacesContextExImpl.java:928)
com.ibm.xsp.util.FacesUtil.convertValue(FacesUtil.java:1152

 So the users were not able to even see what was in that email.

Workaround

Custom ReadOnlyRteRenderer with improved error handling. This way we can adjust HTML if there is an error - but it's really a try-and-guess approach to find the problem.

public class ReadOnlyRteRenderer extends com.ibm.xsp.renderkit.html_basic.ReadOnlyRteRenderer  {

	@Override
	protected String encodeText(FacesContext paramFacesContext, UIComponent paramUIComponent) {
		try {
			return super.encodeText(paramFacesContext, paramUIComponent);
		} catch (Exception e) {
			MailUtil mailUtil = new MailUtil();
			Object v = FacesUtil.getValue(paramFacesContext, paramUIComponent);
			String str2 = v.toString();
			return mailUtil.sanitizeHtml(str2, true, e);
		}
		
	}
}

and MailUtil extract that does the sanitization - I needed it also when we are copying the message content between documents, so it's called in different scenarios, triggering the ACF internally. It's trying to remove the invalid data and if even this would fail, it removes all base64 encoded images.

public String sanitizeHtml(String html, boolean errorCase, Throwable e) {
	FacesContext fc = FacesContext.getCurrentInstance();
	if (!errorCase) {
		try {
			html = ((FacesContextEx)fc).filterHtml("acf", html);
			return html;
		} catch (Exception e1) {
			return sanitizeHtml(html, true, e1);
		}
	} else {
		try {
			if (e!=null && findCause(e) instanceof ArrayIndexOutOfBoundsException) {
				//check for a known problem with base64 followed by cid
				String pattern="<img.*[\"'].*base64,cid[^>]*>";
				html = html.replaceAll(pattern, "");
				html = ((FacesContextEx)fc).filterHtml("acf", html);
				return html;
			}
		} catch (Exception e2) {
			//remove all base64 imgs
			try {
				String pattern="<img.*[\"'].*base64[^>]*>";
				html = html.replaceAll(pattern, "");
				html = ((FacesContextEx)fc).filterHtml("acf", html);
				return html;
			} catch (Exception e3) {
				OpenLogItem.logError(e3);
			}
		}
	}
	
	return "ERROR loading message content due to invalid structure - please report this issue, including link. Message still can be accessed via Notes client"; 

}
	
private static Throwable findCause(Throwable throwable) {
	Objects.requireNonNull(throwable);
	Throwable rootCause = throwable;
	while (rootCause.getCause() != null && rootCause.getCause() != rootCause) {
		rootCause = rootCause.getCause();
	}
	return rootCause;
}

then just register the renderer

<renderer>
  <component-family>com.ibm.xsp.InputRichText</component-family>
  <renderer-type>com.ibm.xsp.InputRichText.ReadOnly</renderer-type>
  <renderer-class>custom.theme.renderkit.ReadOnlyRteRenderer</renderer-class>
</renderer>

We needed it only in read mode as all editable documents would contain the already sanitized version of the data.

The app was not crashing anymore, but another problem was reported.

Unreadable emails after ACF processing

Many systems rely on formatting within the message to provide a nice rendering. Especially when data is mostly stored in a table, it can be really hard to read when ACF stripped out all the styling, except some basic inline styles. It resulted in tables with no borders, so for example, if you have some request forms with many cells, it made them almost unreadable.

Standard mail clients provide for such scenarios buttons like "Allow external content" or "Open the message in a separate window". But this did not make much sense in the app as it would potentially cause all the security issues that ACF was supposed to solve - the page would have been loaded from the same host.

We could possibly try to tweak ACF configuration, but as it's not part of the db template and would have to be deployed (and versioned) separately to all the servers, we decided to not follow this path.

Workaround
Disabling ACF - not the nicest solution, but with we did a few additional steps - see below.

Conflicts between message content and the app

One of the nice things about ACF is that it removes all the active content that can potentially interfere with your app. All JavaScript code, all globally applied styles. Once we disabled the ACF and tested many emails that are processed in the app we started to see weird issues - menus to rendered correctly, buttons not working on some emails. 

When disabling ACF we already thought about some options for isolation. Moving the content into a separate iframe would solve many of the problems, but doing it manually is quite messy. Luckily this is exactly what CKEditor is doing (together with other things like disabling injected JavaScript). 

Solution
Use CKEditor even in read mode. So instead of using a custom renderer we've switched to the standard RT renderer even for read mode.
<renderer>
  <component-family>com.ibm.xsp.InputRichText</component-family>
  <renderer-type>com.ibm.xsp.InputRichText.ReadOnly</renderer-type>
  <renderer-class>com.ibm.xsp.renderkit.html_extended.RichTextRenderer</renderer-class>
</renderer>
We already had some toolbar customizations, so to make sure we don't show any useless buttons, we had to adjust the customization.
<xp:dojoAttribute name="toolbar">
						<xp:this.value><![CDATA[#{javascript:if (pageController.isEditable()) {
 return '[["Format", "Font", "FontSize"],["Bold", "Italic", "Underline", "Strike", "-", "TextColor", "BGColor", "-", "JustifyLeft", "JustifyCenter", "JustifyRight", "JustifyBlock", "NumberedList", "-", "BulletedList"],["Indent", "Outdent"],["Subscript", "Superscript"],["RemoveFormat", "-", "MenuPaste", "-", "Undo", "Redo", "Find", "LotusSpellChecker", "-", "IbmImage", "Table", "Link", "Flash", "-", "PageBreak", "HorizontalRule", "SpecialChar", "Blockquote", "Smiley", "ShowBlocks"],["Maximize", "Source"]]'
}
return '[["Maximize"]]'}]]></xp:this.value>
</xp:dojoAttribute>
Now we have a nice editor (with maximize button) even in read-mode. It provides content isolation and together with antivirus and antispam filters on mail gateways we can be pretty safe. 

But it's still not the end of the ACF story.

Content not being uploaded after switching to edit mode via partial refresh

Emails were working nicely, but when the main document was opened for reading and then switched to edit mode via a partial refresh the changes were not saved. There was no error or warning, nothing. Then the document was opened directly in edit mode, or reloaded with a full page reload, everything worked fine.

As I've already briefly mentioned above the CK Editor is an iframe, so to submit the changes to the server some client side code must take the value from the iframe and put it somewhere into the current form. This is handled by XSP object querySubmitListeners.

There is a different querySubmitListener rendered when the RichText is rendered in read mode. Unfortunately it uses the same id and the XSP object code does not update/replace the listeners.

Snippet from XSP object source:


Workaround
The only way to fix this was to remove the existing listener during the partial refresh before XPages try to add the new one. Another option would be to change the behavior in XSP object, but I did not want to fight with it this time...
<xp:scriptBlock id="scriptBlock3">
	<xp:this.value><![CDATA[
	try {
		if(XSP) {
			if (XSP.querySubmitListeners) {
				var listeners = XSP.querySubmitListeners;
				var listenerId = '#{javascript:getComponent(compositeData.inputId).getClientId(facesContext)}_rteSubmit'.replaceAll(':','_');
				for(var i=0;i<listeners.length;i++) {
					if (listenerId == listeners[i].scriptId) {
						//remove the listener and return as there can be only one
						listeners.splice(i,1);
						break;
					}
				}
			}
		}
	} catch(e) {
		console.error(e);
	}
 ]]></xp:this.value>
</xp:scriptBlock>
The snippet uses compositeData.inputId that is used as the id for inputRichText control.

Conclusion

With these fixes and workarounds, the message content is displayed mostly fine. CK Editor in read mode provides good isolation of message content. There were other problems with images and attachments, but I'll save this for another post.


Labels:

Comments

Post a Comment

Popular posts from this blog

XPages Date Field Issue: Solving the One-Day Jump on Every Save

 A user reported a very strange issue - when a document with a date field is saved, it changes the value one day to the past. With every save. But only for some dates, not all. It turned out to be a mystery that goes deep into XPages and Notes/Java APIs. I've posted a sample on OpenNTF Discord and Serdar tried it on his server - no issue. But he uses the GMT zone and I have CET (Windows set to UTC+1 - Amsterdam, Berlin... to be precise). To cut it short, the issue is caused by daylight saving interpretation between Notes and Java. The date fields (because XPages have no notion of real date-only fields) are stored with 00:00 time component and for some dates the conversion back to Java Date resulted in 23:00 on the previous day. XPages that get the date component as String for the input field, which is then saved back as a previous day during document save. The app is full of date fields and I couldn't add custom logic to every save operation, so I tried to fix it at XPages conv...
Post a Comment
Read more

HCL Domino 12.0.2, Engage 2022 and HCL Factory tour Milan

 I haven't published my recap after Engage this year and the recent HCL Factory tour in Milan is a great opportunity to write a summary about what's happening in HCL (mostly Domino) space. It's a mix of news about 12.0.2, future directions, and my impressions, so it can be a bit chaotic, but I got the impression that many people see it similarly.  Engage 2022 Engage 2022 was great (as always). I love the atmosphere in Brudges. I visited it once after Engage a few years ago and I was happy to come back. This was also the first time I had the opportunity to speak at Engage, which obviously made it a bit more stressful, but also more fun. Together with Domino Jams, HCL continued conversations with customers and partners about the future of their products at Engage. Many of these ideas were now discussed in greater detail in Milan, some of them were even demoed.  My main takeaways from Engage were: Nomad (web and mobile) are a great addition to Notes family Restyle is a great...
Post a Comment
Read more

XPages EL/class-loader memory leak (now with solution)

 We have recently experienced OutOfMemory crashes of XPages app server. The server was recently upgraded to 12.0.1FP1, but we were getting some panic crashes in HTTP even before the upgrade (it was 9.0.1FP10). Our hopes were that the upgrade would stabilize the server, but it's not the case. At least now I start to see what's the problem.  update 8.12.2022 There were actually 3 different leaks. I have rewritten the article to be a bit more clear. I also re-run some of the tests on 9.0.1FP10, so I assume the problems are also in earlier versions. Problem 1 The server is hosting over 1000 NSF sharing the same design + some other custom apps. Not all NSFs are used via web as the app still has classic Notes UI in parallel, so it's a bit tricky to estimate the load. By using tell http xsp show modules I usually see around 350 NSFs active. We kept the default application timeout that should provide reasonable application recycling if it's not used continuously.  We started to...
Post a Comment
Read more