More progress on my Dynamics 365 (CRM) Metadata DIFF tool

I worked on one of my rainy day projects last weekend... a diff tool for Dynamics 365 CRM orgs.

The goal of the solution is to be able to view the major differences between the same CRM org over time or two different orgs (online or on-prem).

The components include two Azure WebJobs and a SQL Server database. One WebJob queries the CRM orgs for metadata (see list below) and the other is responsible for the diff operations and reporting the differences.

So far, the tool compares the following metadata and entity-based data:

• Entity
• Attribute
• OptionSet (Global and entity-based)
• One-to-many relationships
• Many-to-many relationships
• Business Unit
• Connection Role
• Field Permission
• Mailbox
• Organization
• Plugin Assembly
• Plugin Type
• Process
• Process Stage
• Publisher
• Queue
• Role
• Saved Query
• Saved Query Visualization
• Sdk Message Processing Step
• Sdk Message Processing Step Image
• Site Map
• Solution
• Solution Component
• System Form
• System User
• System User Roles
• Team
• Template
• User Form
• User Query
• User Query Visualization
• User Settings
• Web Resource

Next up for the tool is to make the deployment faster and easier. Currently, it's necessary to manually run T-SQL DDL scripts to create the database tables and stored procedures. The plan is to make that a single click using a simple ASP.NET app.

If your organization has the need for an automated CRM metadata tracking or diff process, please get in touch and I'll share some of my experiences in building the tool or, through Altriva, we can talk about sharing the code with you.

Twitter noise-reduction solution in Microsoft Azure is working well

I've been running my Azure-based Twitter "noise-reduction" solution for a few days now. Below are some observations.

First, a recap of what the solution is doing...

The solution runs in Azure to help identify new and unique tweets, eliminating retweets, duplicates and unwanted "noise" tweets by keyword. It sends the tweets that pass the filters to an email folder. I then review the tweets (about 75/day related to Dynamics 365 CRM and Azure) and drag them to another folder. An Azure Logic App reads from that folder and posts the tweets to this blog. So, it's mostly automated, and becomes better over time as it catches duplicate tweets.

This screenshot (Azure Portal) shows the solution services. The solution includes three Azure Logic Apps, a Function App with one function and Azure Storage queues and tables.

Why do this? Mostly because I want to continue to learn how to use Azure Logic Apps, Azure Functions, Microsoft Flow, etc. Second, I have $150 in Azure credits and this seemed like a good use of some of them. And lastly, part of my job is to keep on top of what's happening with Dynamics 365, so creating an intelligent Twitter spy tool seemed like a reasonable way to meet that goal as well.


Lessons learned so far:

• Cost: It costs about $0.75/day to run this solution in Azure. Besides the Azure services shown in the screenshot above, I have two Microsoft Flow workflows querying Twitter for the following hashtags.
• #msdyncrm OR #dynamics365 OR #msdyn365 OR #dynamicscrm OR #dyn365 OR xrmtoolbox 
• #azurefunctions OR #logicapps OR #webjobs OR #powerapps OR #microsoftflow OR #azuresql OR #sqlazure OR @logicappsio

So, it's a relatively inexpensive way to save me a lot of time reading about new CRM and Azure happenings.

• Part of the solution is for a Logic App to auto-send a tweet about the new "Favorite Tweets" blog post. After a few hours of that tweet, each page is getting about 50 unique visitors, so the automated solution is leading to a respectable number of page views. That's not my goal, but if I ran a business then this solution could be used to provide timely and relevant content to site visitors.

• It's reliable. The Azure Logic Apps, Functions, etc. that I created for the solution have run on-time and without fail each day. I didn't really expect them to fail, it was just reassuring to see all green checkmarks in the runtime history.

I'm not sure what my next rainy day side project will be with Azure but I'm sure I'll come up with something soon.

Building a String in an Azure Logic App

In an Azure Logic App, one way to build a string containing multiple values (for example, within a "For Each" loop) is to use an Azure Storage Blob. First, add an action that creates a blob. Then, in your For Each block, use a "Get blob content using path" action along with an "Update blob" action.

To make this work in a loop, though, you'll need to add the following property and value to the For Each block definition in the underlying Logic App code:

"operationOptions": "Sequential"

Example:

"forEach_email": {
       "type": "foreach",
       "foreach": "@body('email_filter')",
       "operationOptions": "Sequential",
       "..."
}

Without that setting, the blob you're building will very likely be incomplete. That's because Logic Apps run loops in parallel. So, by the time the app has retrieved the current blob contents and has appended the next value, another thread of the app has done the same thing and the result will be an unpredictable set of data.

The following For Each block from a Logic App demonstrates this technique.

Note that it's certainly possible that I missed a feature in Logic Apps that helps to build a string, such as some sort of "Append to a String Variable" action, which will make my use of a blob for this look silly. If I did miss this, please leave a comment to set me straight. Otherwise, Logic Apps team: We need an "Append to a String Variable" action.   :)

Taming Twitter with Microsoft Azure

Twitter is noisy.

At least 50% of tweets are duplicates, retweets or auto-generated from old content. And another 25% are likely tweets that, if you're busy, you don't really care to see at the moment. Thus, trying to use Twitter as a source for learning and new information is often time-consuming and not worth the effort. For example, on the topic of Dynamics 365/CRM and Azure, people and bots around the world send out over 1,000 tweets per day -- far too many for most of us to view.

Twitter is useful.

On the other hand, there are tweets that are important (or at least interesting) to see. It's a great resource for keeping up-to-date on just about any topic.

Twitter can be tamed.

This blog post demonstrates one way to tame Twitter: To cast aside unwanted or duplicate tweets to get to the good stuff.

The solution described below utilizes various Microsoft Azure and Office 365 services to achieve the following goals:

• Receive Tweets by e-mail that match a keyword or hashtag
• Eliminate duplicate tweets, retweets and bot tweets
• Eliminate tweets that contain a particular keyword (e.g., #jobs)
• Categorize tweets within e-mail folders
• Benefit from Twitter by significantly reducing the "noise"
• Have full control over the tweet filtering process to further enhance the solution

Get Ready to Build

If you're like me and find benefit in using Twitter but find it too time-consuming to wade through the noisy tweets, and if you're a DIY type of person who would rather build a solution then buy one, then this tweet tamer solution will be worth the effort.

Solution Components

This solution utilizes the following components:

• Microsoft Azure:
• 1 Logic App
• 1 Azure Function
• 1 Azure Storage Queue
• 2 Azure Storage Tables
• Microsoft Office 365
• 1+ Microsoft Flow workflows
• Outlook (desktop or web)

Time to implement: About 1 hour

Cost per month: It depends. If your Twitter searches are relatively narrow, running this solution might be free or not more than a few dollars per month. To avoid a big bill, avoid searching Twitter for keywords or hashtags (in your Flows) that will return hundreds or thousands of tweets per query.

Prerequisites

• Microsoft Azure Subscription
• Microsoft Office 365 account
• Optional: Microsoft Flow (Can also use a Logic App in place of Flow)
• Some software development or scripting experience will be useful but it's not mandatory
• Download and install Microsoft Azure Storage Explorer

Solution Runtime Overview

In a nutshell, this tweet-taming solution works in this way:

1. Microsoft Flow monitors Twitter for specific hashtags, keywords or phrases and posts matching tweets to an Azure Storage Queue.
2. Azure Logic App reads messages from the queue every 30 minutes, sends tweet messages to an Azure Function for analysis and storage.
3. The Azure Function determines whether the tweet is a duplicate or is an unwanted tweet by keyword. It does this by storing and matching tweet messages with Azure Storage Tables.
4. The Azure Logic App conditionally sends the tweet message to an email account. Office 365 Email rule moves tweets to specific folders based on keywords.

Note: In addition to Microsoft Flow, it's also possible to retrieve tweets with an Azure Logic App and other means (e.g., Twitter API). I chose to use Flow for this solution because it's easier to add additional tweet retrievers with Flow compared with creating separate Logic Apps to capture tweets. Plus, Flow is included with my Office 365 account, so using it does not use up my Azure Subscription credits.

Implementing the Solution

This section provides the steps for creating this tweet-taming solution.

Note: Since the procedures for creating and managing Azure services changes often, and since there are lots of online resources available, I am not providing complete step-by-step instructions. For example, when I say to create an Azure Storage account, I don't provide the specific steps for doing that, but I believe I've provided sufficient instructions to understand and build the solution. It just might take some research on your part to complete a step that you haven't done before.

Create an Azure Resource Group for this solution

In the Azure Portal, start by creating a Resource Group in Azure to assign to the services that you'll be creating in the steps below. Using a Resource Group helps organize Azure services that are part of the same solution.

Create the Azure Storage Queue and Tables

1. Create an Azure Storage account.
2. In Microsoft Azure Storage Explorer (or similarly capable tool), create a queue named "tweets" and two tables, one named "tweets" and the other named "unwantedTweetKeywords". There's no need to add additional columns to the Tables.

Create the Azure Function

1. Get the Azure Function files from the GitHub Gist location.
2. Create an Azure Function App. Name it something like "TweetTamer".
3. Create the Azure Function. Use the GenericWebHook-CSharp template. Name it "ProcessTweet".
4. Create the Azure Function files to correspond with the files you retrieved from the provided Gist location. Your function should have three files: run.csx, function.json and project.json.

Create the Azure Logic App

Create the Logic App as shown in the provided screenshots. Name it "ProcessTweets" or any name that you choose.

The "ProcessTweet" action shown below is where the Logic App calls the Azure Function. Construct the JSON string to pass to the function. If the return value from the function does not contain "ignore_tweet" then the tweet text is sent via email. The last step removes the Azure Storage queue message.

Create Rules In Outlook to Move Tweets

The solution is almost ready for use. One optional but recommended step is to create one or more email routing rules in your email application to move emails that start with "New Tweet" to a particular folder. Doing this will help keep your inbox clear of tweets and will help categorize the tweets to further put them into context.

Create Microsoft Flow workflows to receive Tweets and Send to Queue

The last step is to create one or more Microsoft Flow workflows to periodically query Twitter and send tweets to the Azure Storage queue that you created in a previous step.

1. Login to Microsoft Flow
2. Create a blank Flow.
3. Search for "Twitter" and select the trigger named "Twitter - When a new tweet is posted"
4. For the search text, enter one or more hashtags or keywords separate by the word "OR". Example: #msdyn365 OR #msdyncrm OR #dynamics365
5. Click "Next Step" and search actions for "azure queues". Select the "Azure Queues" action then select "Azure Queues - Put a message on a queue".
6. For the queue name, enter "tweets". For the message, select the "Tweet text" option from the dynamic content list. If you want to include a URL to the tweet, then see the image below for an example of how to construct the URL.
7. Click "Create flow" to save the flow and then click "Done".

Twitter is Tamed!

Within 15 minutes of creating one or more of the Microsoft Flow workflows that retrieve tweets and sends them to the Azure Storage queue, you should start receiving tweets in email.

Over time, the number of tweets you receive will decrease because you will not be receiving as many duplicate tweets, no retweets and no tweets that contain the keywords you specify (see "Ongoing Maintenance" below for details).

The end result is that you can now benefit from Twitter without the noise!

Here's my own tweet about this blog post in Outlook online...

Ongoing Maintenance

Ignoring Tweets by hashtag or keyword

As you receive tweets in email, when you see a word or hashtag appear that you'd like to ignore, you can add the word or hashtag to the Azure Storage table "unwantedTweetKeywords".  For the PartitionKey, enter "Keyword" and for the RowKey, enter the word or hashtag. Since Azure Storage tables don't allow the pound character ("#") in the RowKey column, use "HT" instead. For example, to ignore tweets that have #jobs in the text, enter "HTjobs" into the RowKey column. The Azure Function provided in this solution converts "HT" to the "#" character.

Other Ideas

Below are some other thoughts on using and extending this solution.

• Within your email client, create a folder named "Tweets to Read" and use it to store tweets (links) that you want to review later.
• Also within the email client, one idea is to create a folder for tweets that you want to share with others. You can set up a Microsoft Flow workflow to monitor a "To Share" folder and post the tweets on Slack or Yammer.