The transactional outbox pattern leverages database transactions to update a microservice's state and an outbox table. Events in the outbox will be sent to an external messaging platform such as Apache Kafka. This technique is used to overcome the dual-write problem which occurs when you have to write data to two separate systems such as a database and Apache Kafka. The database transactions can be used to ensure atomic writes between the two tables. From there, a separate process can consume the outbox and update the external system as required.

Fueled by vast data volumes and powerful computing, AI is revolutionizing work. To capture the value of Generative AI for business, companies need to customize LLMs with their enterprise data. But feeding sensitive data into externally hosted LLMs poses security and exposure risks, and self-hosting LLMs carry a heavy operational burden from maintaining complex environments.

Learn how Fivetran’s automated data movement platform allows you to accelerate building Gen AI applications for customer service in Google Cloud with BigQuery and Vertex AI. Kelly Kohlleffel steps you through creating four connectors to BigQuery, including a relational database connector plus Jira, Slack, and Zendesk connectors. Then you’ll see how easy it is to quickly build two Gen AI apps, one for search and one for chat, using Vertex AI and the new customer service datasets in BigQuery.