Automatically Track All Incoming and Outgoing Email Messages in Dynamics 365 without opening Outlook and across any device :: Pure Exchange and Dynamics 365 Server Side Synchronisation

Sometimes organisations want to track all incoming and outgoing emails for a number of users at Server Side without having the user to do anything manually and across all devices. The requirement here is to save the user time from clicking on “Track” emails when they are sending them or having to manually move incoming emails into a tracked folder to be tracked. They also want this to work on every email sent from any device and every email received even if Outlook is not open. This what I call “pure Server Side Synchronisation”.

As the name gives it away, Dynamics 365 Server Side Synchronisation and Dynamics 365 App for Outlook can help us achieve this requirement with some help from Exchange Mail Flow Rules that uses the transportation layer directly. Just to re-iterate, we are here using Dynamics 365 App for Outlook (not the client). If you are not sure what is the difference between Dynamics 365 App for Outlook and the Dynamics 365 for Outlook (also known as the Outlook Client), you can refer to this comparison: https://docs.microsoft.com/en-us/dynamics365/customer-engagement/outlook-app/v8/deploy-dynamics-365-app-for-outlook

 

My approach is using Server Side Synchronisation between Dynamics 365 Online Cloud and Exchange Online but the same approach may work with other setups to achieve the same requirement: Track all incoming and outgoing emails automatically from any device. I’m also applying all of this on 1 single “test user” but you can apply this on as many users as you want. My test user is called “sales test”.

In this post, I will not take you in detail through the process of setting up server side synchronisation. You can refer to the official Microsoft Documentation and this comprehensive step by step guide to set server side sync up first before you continue: https://docs.microsoft.com/en-us/dynamics365/customer-engagement/admin/connect-exchange-online

Once you do all the steps in the above guide and you have fully setup server side synchronisation, you will then need to make sure that your test user has setup folder level tracking in their Outlook (Office Outlook or Web rules are fine). To do this, you can follow the step by step guide here: https://docs.microsoft.com/en-us/dynamics365/customer-engagement/admin/track-outlook-email-by-moving-it-tracked-exchange-folder

In summary, before you automate the tracking of all email messages both incoming and outgoing, you must have Server side synchronisation setup, push (i.e. remotely install) Dynamics 365 App for Outlook to your user(s) and you must also setup 1 folder for Dynamics 365 tracking in your user Outlook. This can be a folder under their Inbox which is then configured in Dynamics 365 for tracking (as per the above guide). For your convenience, I have summarised here the steps to setup Folder tracking (refer to the above guide for detailed guidance):

To Set up a tracked folder

  1. In the User “Personalisation Options” or “Personal Option”, click the Email tab, and then under Select the email messages to track in Dynamics 365, select “All Messages”. 
  2. Then Click on Configure Folder tracking Rules. In the Folder-Level Tracking dialog box, under Exchange Folder, click + New Folder Mapping, click the down arrow in the box that appears, and then select the folder you want to track.Where “CRM Tracking” is a folder under Inbox in Outlook, as shown in the following screenshot:

With that, you have Server Side Synchronisation and Folder Level tracking setup for 1 single folder called “CRM Tracking”. Any email message inside this folder will be tracked. Next, we will setup the automation of the tracking of all incoming and outgoing emails, which is the purpose of this post.

To automate the tracking of all incoming email messages, you will need to create an Outlook email rule to copy all email messages received in the user inbox to the “CRM Tracking” folder. Please remember all your outlook email rules must be server side for this to work. If the rule has “Client-only” next to it, emails are moved between folders client side only and not server side which means tracking won’t work.

So, the first rule to create on Outlook is a rule to copy all messages received to the tracked folder. This rule is simple and will look something like this:

 

then

This rule will copy every received message to the “CRM Tracking” folder. But how about sent messages (outgoing email)? Here is the complex part – so follow me to the letter on this one:

If you were to create a new rule via the “Apply Rule on messages I send” that copies every email sent by the user to the same folder “CRM Tracking”, the new rule will always be “Client-only”. This means sent emails are copied on client side only to the tracked folder but not server side – so nothing will appear in Dynamics 365 (remember this is server side tracking not client side).

Also, I have been told that you can create another rule that starts with “Move messages from someone to a folder”, select the same user of this inbox as that “someone” and the “CRM Tracking” folder as the target and this could copy sent emails to the tracked folder server side. I have tried this but it didn’t work for me – emails were not tracked. If you tried it and it worked for you, then please let me know what you did.

On the other hand, if the user was to cc themselves in every email message they send, they will receive a copy in their Inbox, which will then be moved to the tracked folder – hence, this message will be tracked in Dynamics 365. However, the email message will be tracked as “Received” not “Sent”. This also means the user will have a copy of every message they send in their inbox and while they can automate the “Cc” of themselves, it looks odd to the receiver of the email to see the user cc’ing themselves. Hence, this is not a good solution either – it’s a possible solution but not so elegant.

So here comes the approach that is complex, elegant and guaranteed to work – I have tried it several times by now and it always works.

Thanks to an excellent tip from my friend and fellow MVP, George Doubinski, I used Exchange Mail Flow to Bcc every sent message our test user sends. Here is how to do it:

 

  1. Go to the Exchange Online Admin Centre (also known as Exchange Control Panel). You can jump directly to it via this link: https://outlook.office365.com/ecp (you need to be Office 365 Admin)
  2. Click on “Mail Flow” from the left hand menu:
  3. Under “Mail Flow” è Rules, click on “+” to create a new rule that does the following:
    1. Bcc every email message sent by this user to “itself”
    2. Set a message header to every sent message with value “crmtrack” or “techlabslondontrack” (or any message header value of your choice)

Your rule will look something like this:

Once you create your rule, you will need to go back to Outlook rules and firstly, update the first rule we created that copies all messages from inbox to the tracked folder, and add an exception to avoid messages that has “crmtrack” in the header. This way, all sent messages that have been Bcc’ed and arrive in the user inbox, will not be copied to the tracked folder. This rule now looks like this:

Instead, we will create a 2nd Outlook email rule to move (move not copy) all messages that arrive in the user inbox and which has the header “crmtrack” to the tracked folder. This rule will look like this:

 

So to summarise, we ended up creating 3 rules in total:

 

  1. Outlook Rule to copy all email messages that arrive in the user inbox to the tracked crm folder except if the message has “crmtrack” in the message header.
  2. Exchange Online Mail Flow rule that works at the transportation layer to “Bcc” every single email sent from the user to themselves and adds the “crmtrack” header value to the sent email message header
  3. Outlook Rule to move all email messages that arrive in the user inbox that has the “crmtrack” header to the tracked folder.

The reason we use copy for all incoming emails is to allow the user to continue to use their inbox as normal keeping all received emails in inbox. The reason we use move (not copy) for sent emails that arrive as Bcc in the user inbox, is to make sure that all sent emails do not stay in the user inbox. They will continue to exist in the sent folder and a copy will be sent to the tracked folder.

 

The result of all of the above complicated approach is that ALL incoming emails are tracked in Dynamics 365 automatically as “Received” emails and ALL outgoing emails are tracked in Dynamics 365 CRM automatically as “Sent” emails. As this approach is using Exchange Online transportation layer, this is a pure Server Side Synchronisation of Email messages with Dynamics 365 which means it will work with “ALL Emails” sent from ANY Device and any app as the synchronisation happens at the server side and not on the client side. So if you sent an email from the Outlook for the Web (Web Mail), sent it from an Android device email client, iPhone Mail app, Outlook app on iPhone, Office Outlook or any other medium, all emails sent and received are tracked in Dynamics 365.

This is how it looks like in Dynamics 365: Tracked emails sent from any device and emails received while Outlook is closed:

Thanks for reading thusfar! I have spent a lot of time researching and writing this up with the aim to help anyone who is trying to achieve full Dynamics 365 incoming and outgoing email tracking on server side fully regardless of which device or client the email was sent from.

I will be posting another blog post about the same subject soon but this time I will achieve the same result using 100% Exchange Mail FLow Rules via the Transportation Layer.

I hope this helps!

New free XrmToolBox Community Plugin: UCI Dashboard Reset

As a new contribution to the Dynamics 365 community, I have just released a new free XrmToolBox plugin that is simple but I believe is important and much needed for Unified Client Interface (UCI) Dashboards.

Currently, if a Dynamics 365 user sets a Dashboard as their default, they can’t change this back to none. They can change their default dashboard but it will always have a value. It cannot be set to “none” without writing some code to reset it.

The implications of this is that if you have multiple dashboards that you want to display as separate links in the Sitemap of any UCI App, users with default dashboard set, will always go to their default dashboard regardless of where this link is pointing to.

For example: if you have in the sitemap a sub-area / Link pointing directly to dashboard 1 and another sub area pointing directly to dashboard 2, then a user with a default dashboard set to dashboard 3, will always go to their default dashboard (dashboard 3) every time they click on dashboard 1 or dashboard 2.

This is because UCI apps will always prioritise default dashboard over the direct link to a specific dashboard.

 

The solution: Re-set default dashboard to none for some / all users. This can only be done via code but now it is a simple click in our free XrmToolBox plugin.

 

This plugin is offered free as a community solution delivered to you by Mohamed Mostafa and Mark Alber from TechLabs London.

 

Please leave a comment below and/or give us a positive review on XrmToolBox website if you found this XrmToolBox plugin helpful.

 

Thanks!

Disappearing Record Level Team Access security permissions and Manual Record Shares Dynamics 365 v9

Our team at TechLabs London have recently faced a fairly complex and bewildering issue on Dynamics 365 v9. We have setup access team templates against the Contact entity to manage access to Contact records individually. If you haven’t used Access Teams before, you can read more here, but in essence it is a way to control which user can see which contact at a record level.

Our customer has converted from another vendor (a.k.a SF) to our iProperty Cloud solution (http://iProperty.Cloud) built on Dynamics 365 v9 and the new Unified Client Interface (UCI). As part of our data migration, we imported thousands of contacts to Dynamics 365 Customer Engagement and allocated a number of users for each contact record based on a predefined list and set of rules. This was a custom data migration process to create these record level access records using our own TechLabs London propriety migration solution.

Once our data migration into Dynamics 365 was complete, we checked Access Team security and everything looked absolutely fine. The next day after going live, Team access security disappeared for few thousand contacts – but NOT all contacts. Upon looking at the POA table (Principle Object Access Table) where security is set, we found that about 4K access records have disappeared. Literally disappeared!

Apologies for the long story but I want you to be aware of everything we tried to resolve this issue as one of these steps may help you fix your issue (which may or may not be similar to ours).

Basically, we spent the following few days trying lots of different resolutions to try to find out the issue.

  1. First we re-imported Access team records using our migration App – everything went back working fine until the next day in the morning – 4K access records disappeared again!
  2. We tried setting the record level security using “Share” not Access Team (manually and programmatically). While we know that Access Teams are “hidden share” and both get created in POA table, we thought this may solve the issue. It didn’t – next day, everything disappeared.
  3. One possible cause of these record level access security issues could be triggered by re-parenting. For example if contacts are associated to an account and you give ownership of this account to a team while the relationship behaviour is set to cascade all, then the child contact records will become accessible to this team. We removed any cascade relationship behaviour – but still didn’t help. Next day, all access disappeared.
  4. We then Disabled all our custom plugins that remotely affect security and access – we had few automation on security between different entities and teams. No luck.
  5. Also we disabled our own security roles synchronisation process (Azure function) which synchronises security between Dynamics 365 and SharePoint security. Nope! Not even that.
  6. We deleted and recreated the team access template then re-imported all access records. Still same issue. All 4k access records were deleted from POA table the next morning.
  7. We setup an hourly extraction routine to extract Access records from POA table to a separate Azure SQL. We monitored and tried to find out the issue or the cause – still nothing.
  8. After blaming ourselves and our code for good few days, we raised a ticket to the Dynamics 365 Support and Product Team.

Finally, the issue was found to be related to the Table: SubscriptionTrackingDeletedObject table and an associated cleanup process that runs every day in the morning (our UK time) in the Dynamics 365 Platform. Apparently this table includes a list of GUIDs for records that need to be deleted from the POA table (and other locations). This table is not exposed via API (or at least we are not aware of a way to).

Basically, it came down to the fact that we imported contacts before go live the first time with these contacts original GUIDs as they were coming from an older CRM system and we wanted to maintain the relationship between entities and records. This is normally fine, however, at one point before going live we had to make some changes to the structure so we deleted those imported contact records and re-imported them with the same GUIDs again. When we did that, apparently these records and all their related security access records were marked for deletion from the POA table. When we re-imported, these contact GUIDs and their related access records were still marked for deletion. Hence, every day in the morning the good old record deletion clean up process SubscriptionTrackingDeletedObject would delete our access records and drive us crazy.

The fix was a script created and run by the Dynamics 365 support team which permanently fixed the issue for us but not before we have learnt a few hundred ways of how to try to fix such issues!