Data Streaming

Overview

A core component of building any integration is not only being able to make outgoing actions from your application to a Third Party App but also being able to do the inverse: listen for incoming events from a Third Party App and direct them to your server. We call this streaming data to a destination. If you've read the Quick Start, you're likely already familiar with this concept. Let’s take an example:

Assume you’re building a Hubspot integration. You want to be alerted each time a new contact (i.e. a lead) is created in the end user's Hubspot.

Reminder: Alloy Embedded supports two types of triggers: realtime events and polling triggers. With realtime events (i.e. webhooks), Alloy Embedded registers whichever webhooks you've designated. Our system then proxies the data to you once the event has been received. But not every app supports realtime events (common examples of this include Magneto, Netsuite, etc). To get around this, Alloy Embedded supports polling – recurring jobs that run on our infrastructure to periodically ping the Third Party App's API and checks if anything has been updated or created. While polling events aren’t quite as fast as realtime events they work just as good.

Streaming Data to your Application

To stream data to a webhook URL on your server, select the Data Stream block from the utilities section. You should see an input field to enter your webhook URL as shown in the screenshot below.

Select "Hubspot Trigger" as the "Block to Stream." Then, enter the URL of the webhook on your server you want Alloy Embedded to proxy the data to. This endpoint must support POST requests otherwise the body data Alloy Embedded will send won't be reflected on your server. You can use a tool like ngrok for testing.

Note: Alloy Embedded will send an X-Alloy-Signature header with every request which you can use to verify this payload is coming from us.

Turn on the workflow by activating it! Once a user installs the workflow, every time a new contact is created in Hubspot, Alloy Embedded will send you a webhook.

Streaming Data to a Warehouse

You can also stream data to data warehouses (e.g. Snowflake, Redshift, BigQuery), databases (e.g. MongoDB, MySQL, PostgreSQL), ingestion tools (AWS EventBridge, S3, CloudStorage, Pub/Sub).

Let’s take an example. Assume you want to send Hubspot contact events to your data warehouse, in this example, BigQuery. Alloy Embedded has built in batching and pagination technology to reduce complexity and and make streaming even the largest data sets a breeze. To configure this flow, you’ll add a BigQuery block onto the workflow canvas after the Hubspot event as seen below.

Select "Hubspot Trigger" and then click the top level object. Notice the tooltip indicates that by selecting the "Hubspot Trigger" you're choosing to select the entire block output. In other words, we're telling Alloy Embedded to send the entire raw payload received from Hubspot to BigQuery.

Next, select the “Hide this block from users” toggle to hide BigQuery from the end user. This means Alloy Embedded will use the BigQuery credentials you specified to stream the execution data to your BigQuery warehouse. Note that we did not select this for Hubspot.

That's it. If you'd like to use a different destination, feel free to try review our Destinations Glossary.

Wrapping up

In this tutorial, we looked at how to stream data from an event to a destination. As we've seen, destinations can be either your own server or a Third Party App such as a data warehouse. Alloy Embedded sports dozens of possible destination options including: data warehouses (Big Query, Snowflake, Redshift, etc), databases (MySQL, MongoDB, MariaDB, etc) and other ingest methods (Google Pub/Sub, AWS S3, etc).