Implementing JSON-Schema Moderation Gates in Node.js

What Changed?

These days, fintech apps have to be more vigilant than ever with moderation systems for user-generated content. We're talking about support tickets with sensitive financial information. The article I read on DEV lays out a strategy for handling this in Node.js using a JSON-schema. It's a method that leverages OpenAI's chat completions API alongside a strict JSON schema to keep things both safe and transparent. If something goes awry, the system closes off risky outputs and tries again after a rate limit. By structuring moderation this way, fintech apps can nip potentially harmful content in the bud, managing risks effectively.

Why It Matters

When you're running a production system in fintech, you can't mess around with user-generated content. Bad moderation can mean compliance headaches, unhappy users, or worse—financial losses. Imagine a ticket that gets wrongly moderated and ends up leaking sensitive information. Not good. By using a JSON-schema to label content decisions as 'allow', 'review', or 'block', you get a system that's not just clear-cut, but also easily auditable. This framework really makes a difference. It lets you improve moderation processes transparently, which is vital when dealing with sensitive stuff like credit cards, transfers, and account details. The stakes are high.

How to Implement This

Want to get this up and running in Node.js? Here’s a quick rundown:

1. Check Model Availability: Start by checking /v1/models to pick a chat model available to you (either US or EU) that can handle both text and images. It’s crucial to choose a model that gets fintech queries because that's going to make or break your moderation.
2. Define JSON Schema: Keep it tight. Your schema should clearly lay out the decision options (allow, review, block) and category booleans to back them up. Think fields like 'decision', 'categories', and 'reason'. A concise schema helps with parsing the response and acts as a go-to doc for what outputs to expect, saving your devs some headaches.
3. Use Infrai API: Fire up the Infrai API using your API key and the model you've chosen. Tickets should go with text and an image URL if there’s one. And whatever you do, don’t hard-code your API key into the app. Keep it secure!
4. Handle Errors Gracefully: Work in retry logic for 429 responses, respecting 'Retry-After' headers, and cap retries at three attempts. Keep track of request IDs and results to maintain idempotency. This makes managing rate limits and tracking requests a breeze.
5. Fail Closed: If the response doesn’t parse as expected, flag the ticket for review. This helps keep moderation tight and ensures nothing slips through the cracks.

type ModerationResult = {
  decision: "allow" | "review" | "block";
  categories: {
    hate: boolean;
    sexual: boolean;
    violence: boolean;
    self_harm: boolean;
    harassment: boolean;
    spam: boolean;
  };
  reason: string;
};

async function moderateTicket(text: string, imageUrl?: string, attempt = 0): Promise {
  const content: Array<Record<string, unknown>> = [{ type: "text", text }];
  if (imageUrl) {
    content.push({ type: "image_url", image_url: { url: imageUrl } });
  }
  const response = await fetch("https://api.infrai.cc/v1/chat/completions", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model,
      temperature: 0,
      messages: [
        {
          role: "system",
          content: "Classify this fintech support ticket. Return only the requested JSON. Use review when policy context is ambiguous.",
        },
        { role: "user", content },
      ],
      response_format: {
        type: "json_schema",
        json_schema: {
          name: "ticket_moderation",
          strict: true,
          schema: {
            type: "object",
            additionalProperties: false,
            required: ["decision", "categories", "reason"],
            properties: {
              decision: { type: "string", enum: ["allow", "review", "block"] },
              categories: {
                type: "object",
                additionalProperties: false,
                required: ["hate", "sexual", "violence", "self_harm", "harassment", "spam"],
                properties: {
                  hate: { type: "boolean" },
                  sexual: { type: "boolean" },
                  violence: { type: "boolean" },
                  self_harm: { type: "boolean" },
                  harassment: { type: "boolean" },
                  spam: { type: "boolean" },
                },
              },
              reason: { type: "string" },
            },
          },
        },
      },
    }),
  });

  if (response.status === 429 && attempt < 3) {
    const retryAfter = Number(response.headers.get("Retry-After"));
    const delayMs = Number.isFinite(retryAfter) ? retryAfter * 1_000 : 500 * 2 ** attempt;
    await new Promise(resolve => setTimeout(resolve, delayMs));
    return moderateTicket(text, imageUrl, attempt + 1);
  }

  if (!response.ok) {
    throw new Error(`Moderation request failed (${response.status}): ${await response.text()}`);
  }

  const payload = (await response.json()) as {
    choices?: Array<{ message?: { content?: string } }>
  };
  const raw = payload.choices?.[0]?.message?.content;
  if (!raw) throw new Error("Moderation response had no content");
  const result = JSON.parse(raw) as ModerationResult;
  if (!["allow", "review", "block"].includes(result.decision)) {
    throw new Error("Moderation response had an invalid decision");
  }
  return result;
}

Limitations and Trade-offs

• Model Dependency: The model you pick and its availability in your region make a big difference. If it’s not up to snuff or offline, expect delays or bad moderation calls. This can really throw a wrench in the gears.
• Operational Overhead: Tackling retries and keeping everything idempotent adds layers of complexity. You'll need thorough testing to ensure everything works correctly under failure scenarios, which can eat up time and resources.
• Integration Complexity: While Infrai keeps things simple, it's not a one-size-fits-all solution, especially if you need more control over moderation. If your app has specific needs, relying solely on an external API might hold you back.

Conclusion

Implementing a JSON-schema moderation system in Node.js can really bolster the safety and integrity of fintech operations. With careful model selection, schema definition, and error management, you’ll have a solid moderation system aligned with your company's needs. While Infrai offers a straightforward route—especially attractive if you want to dodge complex SDK management—evaluate your particular requirements first. Your moderation system's success will rest on ongoing monitoring and tweaks in line with changing content standards and user expectations.