Create an Data Pipeline from Webhook to Slack
This guide shows an end-to-end event pipeline that reads an event from a webhook, generates a formatted string, and publishes the result to Slack. While this is a simple example, it has many event notification use cases, such as:
- submission from website forms (via Cloudflare workers or your custom backend).
- activity from e-commerce platforms on purchases and shopping carts.
- notifications from github on your projects' activities.
- alerts from financial products on your transactions.
- notifications from any product that can invoke a webhook.
This pipeline uses the following features:
- webhook: that creates a public API to receive external events, transform them, and publish them to a topic.
- http-sink: that listens to the same topic and publishes them on Slack.
Objective
Show an example of how to build an event streaming pipeline that receives webhook events, transforms the input into a readable form, and generates an alert. We assume the events are generated by a user submitting a form, and we'll format it accordingly.
Prerequisites
- Fluvio CLI installed locally
- Account on InfinyOn Cloud
Step-by-Step
- Create webhook configuration file
- Create http-sink configuration file
- Download SmartModules
- Start Webhook and Connector
- Test Data Pipeline
Create webhook configuration file
Create a webhook configuration file called form-webhook.yaml
:
All versions are marked with x.y.z
. To find the latest version, run:
fluvio hub connector list
fluvio hub smartmodule list
meta:
name: form-webhook
topic: form-events
webhook:
outputParts: body
outputType: json
transforms:
- uses: infinyon-labs/json-formatter@x.y.z
with:
spec:
match:
- key: "/type"
value: "subscribe"
format:
with: "📢 {} ({}) subscribed on {}"
using:
- "/name"
- "/email"
- "/source"
output: "/formatted"
- key: "/type"
value: "use-case"
format:
with: ":confetti_ball: {} ({}) wants to solve the following '{}' use-case:\n>{}"
using:
- "/name"
- "/email"
- "/source"
- "/description"
output: "/formatted"
default:
format:
with: "{} ({}) submitted a request"
using:
- "/name"
- "/email"
output: "/formatted"
The webhook reads the JSON body, applies the json-formatter
smartmodule to generate readable text, and writes the new record to a topic called form-events
. Checkout labs-json-formatter-sm in github for additional information.
Create http-sink configuration file
Create an HTTP source connector configuration file called slack-form-alerts.yaml
:
apiVersion: 0.1.0
meta:
version: x.y.z
name: slack-form-alerts
type: http-sink
topic: form-events
secrets:
- name: SLACK_USER_ALERTS
http:
endpoint: "https://hooks.slack.com/services/${{ secrets.SLACK_USER_ALERTS }}"
headers:
- "Content-Type: application/json"
transforms:
- uses: infinyon/jolt@x.y.z
with:
spec:
- operation: shift
spec:
"formatted": "text"
The sink connector reads from the form-events
topic and uses the jolt
smartmodule to shift the formatted string into a field called text
per the Slack instructions. Checkout fluvio-jolt in github for additional information.
Add Slack webhook token to InfinyOn Secrets
The Slack webhook link is sensitive information, let's add the Access Token part to secret
in InfinyOn Cloud :
$ fluvio cloud secret set SLACK_USER_ALERTS <webhook-token>
Check out Slack Webhooks on how to create the webhook token.
Download SmartModules
Download the smartmodules used by the webhook ad the connector:
$ fluvio hub download infinyon/jolt@x.y.z
$ fluvio hub download infinyon-labs/json-formatter@x.y.z
Check fluvio smartmodule list
to ensure they've been downloaded.
Start Webhook and Connector
Start webhook listener:
$ fluvio cloud webhook create --config form-webhook.yaml
Check fluvio cloud webhook list
to ensure it has been successfully provisioned. In checkout the webhook link that we'll use to test the pipeline: https://infinyon.cloud/webhooks/v1/[token]
Start sink connector:
$ fluvio cloud connector create -c slack-form-alerts.yaml
Check fluvio cloud connector list
to ensure it has been successfully provisioned.
Test Data Pipeline
Use curl
to send a POST request with a fictious user request to our webhook link. In production environments, this iw what a website would send:
$ curl -X POST https://infinyon.cloud/webhooks/v1/<token> \
-H "Content-Type: application/json" \
-d '{ "email": "alice@acme.com", "name": "Alice Liddell", "type": "subscribe", "source": "front-page" }'
The following alert is displayed in Slack:
`📢 Alice Liddell ("alice@acme.com) subscribed on front-page` will show-up in your slack channel.
That's all folks!