> ## Content Index
> Fetch the complete content index at: https://www.sfrt.io/llms.txt
> Use this file to discover other available public pages before exploring further.

# The regexes had a good run
- URL: https://www.sfrt.io/the-regexes-had-a-good-run/
- Published: 2026-09-21T06:04:09.000Z
- Updated: 2026-09-21T06:04:09.000Z
- Description: Hand-written patterns decided which Salesforce errors were noise. Now a classifier does it, and it says how sure it is.
- Author: Martin Seifert
- Tags: Infrastructure

TL/DR: I replaced a SQL table function full of regexes by one API call that returns a category plus the probability that an error notification is actually something interesting to look at.

---

Salesforce can mail automatic error notifications to a (shared) inbox. Those can be quite noisy, as most of them come from the same handful of known (but uncritical) offenders: an external integration that retries on its own, a merge that gets blocked by design or a quick action complaining when someone double-clicks a button. But a few errors are real. For the past couple of years, every newly discovered harmless (regex) pattern meant a code change. Anything matching none of those cases landed on a person's desk to look into.

## What's wrong with the regexes?

Nothing, technically... they work 😅 But they are also just a list of things I have already seen. That's generally the wrong shape for a monitoring problem, because the interesting email is the one not covered by a known pattern. A regex can only answer "matched" or "didn't match", and the later covers everything from "brand new Apex exception, go look!" to "known error, slightly reworded subject line" 🙄

Obviously, I didn't need better patterns, I needed an error category and a number that estimates how critical an unexpected case is. Fortunately, we live in interesting times when it comes to automatic decision making...

## Why not a chat model?

Running each email through an LLM (with structured output) is an option. However, there is Jev. And since we're currently in the middle of its hype cycle, I thought I might as well give it a try...

TypeSafe sells what they call System One models - Jev being the first. It writes no text at all. Given a piece of state and a set of questions, it answers with types and probabilities. I have the workflow, Jev adds the judgment in the middle of it. The answer is returned as a value the flow can work with without further parsing.

There are three primitives: ****Choice** (pick from up to 255 options), ****Score** (a position on an ordered rubric of 2 to 10 levels) and ****Noul** (one yes/no probability).

Classifying text is not a new idea... What makes Jev interesting is really the convenience, speed and quality of getting a useful response 😜

One POST to `https://api.typesafe.ai/v1/systemone`, two questions in the same call:

```json
{
  "model": "jev-latest",
  "state": { "subject": "<mail subject>", "body": "<plain text body>" },
  "questions": {
    "category": { "type": "choice", "instructions": "...", "criteria": { "<label>": "<description>" } },
    "urgency":  { "type": "score",  "instructions": "...", "criteria": [ "<level 0>", "<level 1>", "<level 2>" ] }
  }
}
```

`state` can be a plain string. I pass an object instead, so subject and body stay separate. What comes back is keyed by the ids I chose 😎

[Introduction - TypeSafe AIJev is TypeSafe’s flagship model and the first System One model. Send state and typed questions; get structured answers your code can use directly.![](https://storage.ghost.io/c/7d/94/7d942fe1-7868-4a1a-b2c9-4eb415b1a546/content/images/icon/android-chrome-192x192-933a4ec0-9259-40a2-95af-c580006ac138.png)TypeSafe AI![](https://storage.ghost.io/c/7d/94/7d942fe1-7868-4a1a-b2c9-4eb415b1a546/content/images/thumbnail/image-955e8f72-8f61-4101-b892-6aa3ee9e9c43.png)](https://docs.typesafe.ai/?ref=sfrt.io)

## The category question

Currently, the category labels are the old (just 2 sampled here) regex categories, plus `other`:

```json
"category": {
  "type": "choice",
  "instructions": "Classify this automatic Salesforce error notification email. Pick the one known category that describes ALL errors in the email. Pick 'other' if at least one error does not clearly match a known category, or if you are unsure: a wrong 'other' costs a person a minute of review, a wrong known category hides a real error.",
  "criteria": {
    "Payment provider processing recurring donations": "NPSP error email in which every error is a recurring donation (context: RD) that could not be processed with the message 'unable to obtain exclusive access to this record', and the user who triggered it is the payment provider integration user. The payment provider performed several actions on the same RD at once and retries on its own.",
    "Doubleclick on reference number generator": "Apex exception in SwissRefNumberBatchGenerator with UNABLE_TO_LOCK_ROW. Caused by someone double-clicking the generator button.",
    "other": "Everything else: any error not described above, any email that mixes known and unknown errors, any new Apex exception, flow error or integration failure, and anything you are unsure about."
  }
}
```

The descriptions carry identifying facts rather than topics. If I simply wrote "merge problems", any Salesforce error containing the word merge might get filed under a known category... But if I name the exact error message or code or the Apex class or additional context (like the integration user), the lookalikes stay out. A first round of testing checked precisely this: the same lock error, triggered by a human instead of the integration user, came back as `other`, at 0.73 likelihood. That's what I wanted, since it's a different situation with a different cause.

The last sentence about cost is rather critical in those instructions. A wrong `other` categorization (false positive) costs a person a minute, but a wrong known category (false negative) hides a real error. Writing that asymmetry down adjusts the decision boundary. The mixed-mail rule ("only if ALL errors match") follows the same principal.

And then, after Jev categorizes each email, I ignore the category label entirely 😅 What I actually need is only `probabilities.other` from its response:

```sql
case when coalesce(el.probability_other, 1) >= 0.2   -- tune here!
     then 'raise' else 'mute' end as notification
```

At `0.2` a notification gets raised to the Salesforce admins even when the model favours a known category, provided `other` gets at least a fifth of the probability. Tweaking that number is one `create or replace function` call.

## A second question, for free

The same request also asks for a three-level Score:

```json
"urgency": {
  "type": "score",
  "instructions": "Rate how urgently someone has to look at this automatic Salesforce error notification. Judge only the operational impact described in the email: how many records or users are affected, whether data was lost or left inconsistent, and whether donations, payments or an integration are blocked. A single failed record that the source system retries on its own is not urgent. Ignore how alarming the wording sounds.",
  "criteria": [
    "Routine or self-healing. A single record failed, the source system retries automatically, or the error is cosmetic. Probably can be ignored or reviewed if it repeats annoyingly often.",
    "Worth a look. A recurring job, integration or automation failed in a way that will not fix itself, but donations, payments and data integrity are not immediately at risk.",
    "Needs action today. Donations or payments are failing, an integration is down, data is being lost or left inconsistent, or the same error repeats across many records."
  ]
}
```

Why Score instead of Noul? A yes/no ("does this need attention?") would have been the smallest column, but it collapses "might be worth a closer look" and "fix this right now" into one bucket. However, that middle level is actually what I'm most curious about while still trying to understand how Jev takes decissions 😜 The Score returns a weighted position from 0 to 2 (1.43 is a possible answer), a confidence level and per-level probabilities.

Once again, the last sentence of the instructions matters: Salesforce error emails all sound alarming ("Developer script exception", "unhandled fault", capital letters everywhere) 😅 Asking Jev to estimate impact gives it something other than vocabulary to work with.

## Failing loud

Since the error notifications arrive in a shared Outlook inbox, they get picked up by Power Automate (whatever other flow tool capable of sending POST requests to Jev would do just as well) and each email is turned into a processed row inserted into an Azure Table Storage table. In case TypeSafe ever doesn't return any answer, `Category = 'unclassified'`, `ProbabilityOther = 1`, `Urgency = 2`, `UrgencyConfidence = 0` get stored and the row is certain to get raised.

Why store it in a table instead of immediately raise? Notification fatigue, of course: I bundle all notifications received since the last checkpoint into a single notification and only send it to the Salesforce admins twice per day. Well, only those worth raising, actually 😜

I tested the setup with 11 synthetic mails and all 11 came back correct; the first real one after deployment returned `other` at 0.59\. Since I got access to TypeSafe on Saturday, that's the only real sample I got, so I'll have to keep watching the table for a couple of days. But if that modified workflow works as expected, Salesforce admins now get an indication on whether a notification is actually worth a look, while the noise remains filtered out 😎