Can't work AI Gateway with bedrock

[watany-dev]

Existing documentation URL(s)

Probably a problem with the signing process, this sample doesn't work(gateway/bedrock), IAM hardcoded something strong so it's not an AWS IAM issue and I can confirm that I have sufficient permissions on the AI Gateway (/app).

https://developers.cloudflare.com/ai-gateway/providers/bedrock/

import { Hono } from "hono";
import { CloudflareWorkersAI } from "@langchain/cloudflare";
import { logger } from "hono/logger";
import { streamText } from 'hono/streaming'
import { AwsClient } from 'aws4fetch';

const app = new Hono();
app.use(logger());

app.get("/app/:msg", async (c) => {
    const msg = c.req.param('msg')
    const cloudflareAccountId = c.env.CLOUDFLARE_ACCOUNT_ID
    const cloudflareApiToken =  c.env.CLOUDFLARE_API_TOKEN

    const model = new CloudflareWorkersAI({
      model: "@hf/mistralai/mistral-7b-instruct-v0.2", // Default value
      cloudflareAccountId,
      cloudflareApiToken,
      baseUrl: `https://gateway.ai.cloudflare.com/v1/${cloudflareAccountId}/${c.env.CLOUDFLARE_AI_API_NAME}/workers-ai/`,
    });
    
    const responseStream = await model.stream(msg);
    console.log(responseStream)
    return streamText(c, async (stream) => {
        for await (const chunk of responseStream) {
            console.log(chunk)
            await stream.writeln(chunk)
            await stream.sleep(100)
        }
    })
});

app.get("/gateway/bedrock/:msg", async (c) => {
    const msg = c.req.param('msg');
    const cfAccountId = c.env.CLOUDFLARE_ACCOUNT_ID;
    const gatewayName = c.env.CLOUDFLARE_AI_API_NAME;
    const region = 'us-east-1';

    // AWS Bedrockの正式なAPIエンドポイントURLを使用して署名
    const modelUrl = `model/amazon.titan-embed-text-v1/invoke`;
    const awsUrl = `https://bedrock-runtime.us-east-1.amazonaws.com/${modelUrl}`;

    const awsClient = new AwsClient({
        accessKeyId: c.env.AWS_ACCESS_KEY_ID,
        secretAccessKey: c.env.AWS_SECRET_ACCESS_KEY,
        region: region,
        service: "bedrock"
    });

    const requestData = { inputText: msg };
    const headers = { 'Content-Type': 'application/json' };

    const presignedRequest = await awsClient.sign(awsUrl, {
        method: "POST",
        headers: headers
    });

    // Cloudflare AI Gatewayを経由するURLを設定
    const gatewayUrl = new URL(`https://gateway.ai.cloudflare.com/v1/${cfAccountId}/${gatewayName}/aws-bedrock/bedrock-runtime/${region}/${modelUrl}`);

    // AWS署名ヘッダーを追加
    const finalHeaders = new Headers(presignedRequest.headers);
    finalHeaders.set('Host', gatewayUrl.host);

    const response = await fetch(gatewayUrl.toString(), {
        method: 'POST',
        headers: finalHeaders,
        body: JSON.stringify(requestData)
    });

    if (!response.ok) {
        return new Response("Invalid response", { status: response.status });
    }
    const data = await response.json();
    return c.json(data);
})

export default app

What changes are you suggesting?

I would like a sample code that works well enough

Additional information

No response

[kodster28]

Howdy @watany-dev, the example you provided here has quite a bit more than the example in our docs (combining some of the Workers AI stuff, which is great overall but makes it harder to eval what exactly is wrong).

If you just run the example present in our docs, does that one work? Or does it also error out on you?