Salesforce Syncing via OpenFN
A step-by-step guide on how to connect CommCare and Salesforce! In this how-to guide built by TechnoServe, a nonprofit that works with communities to build competitive farms, businesses, and industries, you will learn:
How to set up a two-way data sync from CommCare to Salesforce via OpenFN.
How to create auto-generated IDs using concatenation.
Background
For starters, here’s a quick review of how TechnoServe is using the data sync and concatenation:
Syncing two-way data via OpenFN
At a high-level, here’s how this sync works for the TechnoServe team:
TechnoServe collects data via CommCare during training sessions.
This data is sent to TechnoServe’s central project monitoring database in Salesforce using a tool called OpenFN as the “middleman.”
Data related to TechnoServe’s projects, trainers, groups, etc., is then stored in Salesforce and sent through OpenFN to CommCare where cases are generated.
TechnoServe’s field staff then use surveys to update cases and report data related to those cases back to Salesforce via data forwarding to OpenFN.
This process ensures that TechnoServe is processing as much data locally as possible, so that field staff always have the most up-to-date information on their mobile devices, even if they are not in a position to sync. For instance, TechnoServe auto-generates participant IDs when they register new participants based on the trainer that registered them, their training group, and the household they were registered in. This means instead of waiting for Salesforce to assign the ID, TechnoServe allows CommCare to do the heavy lifting, and ID’s are available for use in the field without needing to sync.
Using concatenation on auto-generated IDs
First, what is concatenation?
Concatenation is a series of interconnected things or events.
How does concatenation come into play on auto-generated IDs, and why is it helpful?
Many organizations struggle with generating unique, automatic IDs for tracking purposes. After attempting many solutions, such as pre-assigning a block of IDs and QR codes and tasking trainers to assign IDs, TechnoServe wanted to create a solution that was automatic and did not require input from the trainer for cleaner or more reliable data for reporting and analysis. They also wanted a solution that would require less lag time for trainers responsible for registering new participants.
By dynamically assigning participant ID’s in the field directly on the mobile device, even when the trainer is offline, TechnoServe is able to minimize lag time in training sessions. For example, TechnoServe’s training staff are often out in the field for extended periods of time, without reliable Internet access. Without the data sync, trainers would have to sync, wait for the data to make it through OpenFN to Salesforce, then back down to CommCare through OpenFN. While this process does not generally take long, unreliable Internet can pose a serious challenge when trying to receive a message back from Salesforce with the participant’s ID.
At a high-level, how does it work?
TechnoServe pulls appropriate values such as country codes, cohort numbers, training IDS, and household numbers into their CommCare survey prior to creating an ID.
Using case management and a hidden question, CommCare concatenates the values.
This value is saved to a case property when a case is created.
In addition, TechnoServe updates a “household counter,” which is an incremental counter on a parent case that keeps track of the next household number that should be assigned to new participants.
The combination of the two-way data flow between CommCare, and Salesforce and generating unique IDs on field worker’s mobile devices with concatenation, allows TechnoServe to run near real-time reports in Salesforce on things like training attendance, training observations and participant feedback, and demo plot observations.
Step-by-step guide
While the use case specifically associated with this released version may be somewhat specific to agricultural technical assistance, the idea is that other organizations can adopt the same framework and modify as needed to suit their particular needs.
Use the following step-by-step guide to learn how to set-up the two-way sync between CommCare and Salesforce, as well as how to create auto-generated IDs using concatenation in CommCare.
1. The first thing you need is a CommCare account. If you are interested in two-way data flow between CommCare and Salesforce, you will, at a minimum, need the Standard plan to take advantage of the API Access.
2. When you are creating a survey for use with Salesforce, add a hidden question to identify the type of survey that you are creating. TechnoServe uses ‘survey_type’ as the ID for their hidden question, and in the Calculate Condition field they enter something like ‘New Participant’ or ‘Household Survey’ (remember the single quotes). This will help you uniquely identify survey submissions that come from CommCare.
3. Sign up for an account at OpenFN.org. You can get a free account to test out the integration between CommCare and Salesforce.
4. You will then need to copy your OpenFN Inbox URL. You can copy your Inbox URL by clicking on Inbox in OpenFN and then clicking the Copy URL button. We will be sending data from your submitted surveys to your OpenFN Inbox.
5. Once you are using a paid subscription, and have signed up for an OpenFN account, navigate to CommCare, and go to Project Settings > Data Forwarding to set up data forwarding.
6. Click on the + Add a forwarding location under the Forward Forms section of the page. Use the following settings to set up your data forwarding:
Payload Format: JSON
URL to forward to: Your OpenFN inbox URL that you copied in Step 4*
Authentication protocol: None
Username: Leave blank
Password: Leave blank
Include ‘app_id’ URL query parameter: TRUE
* Once you copy and paste your OpenFN inbox URL, click on the Test Link button. You should get back a response below the button that begins with Success! Response is: 200.
7. Once you have entered the settings above, click on the Start Forwarding button at the bottom to begin forwarding survey response.
8. Now you are ready to set up OpenFN. Navigate back to OpenFN and click on the project space that you set up when you signed up for your account.
9.. On the left-hand side, you will see a set of options. The first thing you will need to do is set up Credentials for Salesforce. This will allow OpenFN to access Salesforce and send data from your CommCare survey to specific fields that we set in the job. Click on Credentials and then click on the New button at the top. A drop down menu will appear with all of the OpenFN connectors, select Salesforce. Use the following settings when setting up your credentials, then click the Save button.
Label: Anything you want, we like to use Test Salesforce (for our sandboxes) and Production Salesforce (for our production instances).
Username: Your Salesforce username. Don’t forget, if you are using a sandbox (which we recommend the first time around), append the name of the sandbox to your username (i.e., myusername@example.com.test).
Security Token: Your Salesforce security token. If you have never set up an integration before, or used the old Apex data loader, you may have never come across this before. You’ll likely need to reset your token to get it emailed to you. A quick URL hack to do so is to copy and paste this into your browser’s address bar -- https://[SalesforceDomainHere]/_ ui/system/security/ResetApiTokenEdit?retURL=%2Fui%2Fsetup%2FSetup% 3Fsetupid%3DPersonalInfo&setupid=ResetApiToken -- make sure you change the [SalesforceDomainHere] to your Salesforce domain (for example: na30.salesforce. com). You can login to Salesforce first to see what your Salesforce domain is. Click the Reset Security Token button and then check your email.
Password: Your Salesforce password.
Login URL: Either login.salesforce.com for Production or test.salesforce.com for a Sandbox.
10. The next thing you need to do is set up a “Trigger.” Triggers watch for incoming messages and run them through jobs when they match the filter criteria. First, we need to send a test message to OpenFN from CommCare. Simply complete the survey that you created, navigate to your Inbox in OpenFN, and watch for a new message to arrive.
11. When the message arrives, click on the message, then click on the View/Edit Body button. If you are familiar with JSON, I suggest looking at the Power Editor by clicking the Power Editor button. If you followed our tip to add ‘survey_type’ to your survey in Step 2, then the next step will be easy. If not, I suggest going back to CommCare and adding it now, then submit another survey.
12. See the key, “survey_type,” has a value of New Participant? We are going to use this to write our first Trigger. On the left-hand side of the screen, click on Triggers then click on the New button at the top. Use the following settings, then click Save.
Trigger Type: Message Filter: Job runs when matching messages arrive.
Label: You can put anything you want here, this just helps you identify the trigger later.
Filter Criteria: {“form”: { “survey_type”: “New Participant”}} -- note that this needs to be valid JSON. You can use any number of different criteria in a Trigger, the important thing is that it’s unique from other triggers.
13. Now it’s time to set up your first Job. A job defines the specific series of tasks or actions to be performed when an incoming message that meets our Trigger criteria (survey_ type = New Participant) is received. Click Jobs on the left-hand side of the screen, then click New.
14. Use the following settings when creating your job:
Name: This can be anything you want, it will just help you identify the job later.
Filepath in GitHub: Leave this blank, Git version control is outside the scope of this tutorial.
Adaptor: Salesforce
Version: Latest (auto-upgrade)
Trigger: Select the Trigger that we just created.
Credential: Select the Credential that we created earlier.
Expression: This is where you will spend the majority of your time. If you aren’t familiar with JavaScript, I suggest spending a few hours on one of the free coding websites (codecademy, codeschool, etc.) to learn the basics. The example below is only a small fraction of what OpenFN will allow you to do with JavaScript and the mapping the data to Salesforce, among other data sources.
15. Once you finish writing your job, it’s time to test it out. Click on Inbox in the menu on the left-hand side and locate the “message” that you sent from CommCare. If you wrote your filter correctly, you should see a label next to the message under the Potential column with the name of the job we just created.
16. Click on the label next to your message and the job will run, processing your incoming message. If the job is successful, the message will turn green and display a green check. If the job fails, the message will turn red and display a red ‘X’.
17. Now let’s check Salesforce. Navigate to the object that you mapped in your OpenFN job (in TechnoServe’s case ‘Participant’) and locate the most recent record that was created. You should see the participant that we created via our survey.