ReferenceError in SageMaker invokeEndpointWithResponseStream

[cshen-dev]

Describe the bug

I created an AWS SageMaker endpoint with Mistral Jumpstarter (jumpstart-dft-hf-llm-mistral-7b-ins-20240130).

sagemakerRuntime.invokeEndpoint worked as a charm however when I started exploring its stream capability, I met the issue below.

Error [ReferenceError]: payload is not defined
    at Request.extractData (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/protocol/rest_json.js:78:17)
    at Request.callListeners (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'ReferenceError',
  time: 2024-02-02T00:39:53.693Z,
  statusCode: 200,
  retryable: false,
  retryDelay: 62.00239553897755
}

Expected Behavior

[ReferenceError]: payload is not defined

^ This error should not occur which I guess is caused by mis-mapped data structure between SageMaker endpoint's Response and the sdk client's Request.

Current Behavior

It's throwing error and block the downstream processing

Reproduction Steps

1. Set up a SageMaker endpoint
2. Use the code block below (which is actually from AWS official doc)

// In Node.js, events are streamed and can be read as they arrive.
sagemakerruntime.invokeEndpointWithResponseStream({/** params **/}, function(err, data) {
  if (err) {
    // handle error
    return console.error(err);
  }

  var eventStream = data.Body;

  eventStream.on('data', function(event) {
    // Check the top-level field to determine which event this is.
    if (event.PayloadPart) {
      // handle PayloadPart event
    } else if (event.ModelStreamError) {
      // handle ModelStreamError event
    } else if (event.InternalStreamFailure) {
      // handle InternalStreamFailure event
    }
  });
  eventStream.on('error', function(err) { /** Handle error events **/});
  eventStream.on('end', function() { /** Finished reading all events **/});
});

Possible Solution

No response

Additional Information/Context

No response

SDK version used

2.1549.0

Environment details (OS name and version, etc.)

macos

[aBurmeseDev]

Hi @cshen-dev - thanks for reaching out.

I’m trying to understand if the error you’re getting is actually related to SDK because it seems like a generic JavaScript error. Can you share your code where you’re making client request with SDK? Are you defining the “payload” somewhere?

I’d be happy to further investigate once you share more information requested.
Best,
John

[github-actions]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[inf3rnus]

FYI @cshen-dev I've created a fix for this with #4607, pending Amazon's evaluation of the change.

In the meantime you can pull my fork @ https://github.com/inf3rnus/aws-sdk-js

[rjvgupta]

I had a similar error on this file when using lambda's invokeWithResponseStream. Issue was resolved when making changes as per my pull request (#4609). This is clearly a code issue as there is no variable payload referenced in the file but it is an attribute of the rules object and is being used as such in subsequent conditional blocks (e.g. lines 85,87,89).

2024-03-10T19:54:58.963Z	2a359c5e-a419-406f-9092-84b26d378eeb	ERROR	Error invoking Lambda function: Error [ReferenceError]: payload is not defined
    at Request.extractData (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:78:17)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'ReferenceError',
  time: 2024-03-10T19:54:58.963Z,
  statusCode: 200,
  retryable: false,
  retryDelay: 64.36430770472576
}

[rjvgupta]

Here's a simple repro using 2 lambda functions, with error above, using published SDK. Same code is successful when using lambda layer that has code changes from my pull request.

lambda1:

var AWS = require("aws-sdk")
AWS.config.update({region: 'us-east-1'}); 
var lambda = new AWS.Lambda();

exports.handler = async (event) => {
  var params = {
    FunctionName: 'lambda2', 
    InvocationType: 'RequestResponse',
    LogType: 'Tail',
    Payload: JSON.stringify({message: 'Do Nothing.'})
  };
        
  const result = await lambda.invokeWithResponseStream(params).promise();
  const events = result.EventStream;
  for await (const event of events) {
    if (event.PayloadChunk) {
      console.log(event.PayloadChunk);
    } 
  }
  
  const response = {
    statusCode: 200,
    body: JSON.stringify('Hello from Lambda!'),
  };
  return response;
};

lambda2:

exports.handler = awslambda.streamifyResponse(async (event, responseStream, _context) => {
  responseStream.write('message1');
  responseStream.write('message2');
  responseStream.write('message3');
  responseStream.end();
});

[aBurmeseDev]

Hi everyone on the thread, appreciate all your effort into this behavior.

I was able to reproduce this and will be further looking into it with the team. Will post once we have updates.

[rjvgupta]

@siddsriv / @aBurmeseDev - any update?

[inf3rnus]

@rjvgupta if you're only using this part of the AWS sdk, v3 of the sdk solves this problem, and consuming the output of it is a lot easier.

e.g.

async function callSageMakerEndpoint({ endpointName, body }) {
  const params = {
    EndpointName: endpointName,
    ContentType: "application/json",
    Accept: body.stream
      ? "text/event-stream; charset=utf-8"
      : "application/json",
    Body: JSON.stringify(body)
  };

  if (!body.stream) {
    const { Body } = await sageMakerRuntimeClient.send(
      new InvokeEndpointCommand(params)
    );

    const buffer = Buffer.from(Body);

    const jsonResults = JSON.parse(buffer);

    return jsonResults;
  }

  // streaming output
  const { Body } = await sageMakerRuntimeClient.send(
    new InvokeEndpointWithResponseStreamCommand(params)
  );

  const readStream = Readable.from(Body);

  readStream.on("error", error => {
    console.error(`Sagemaker endpoint error: ${endpointName}`, error);
    readStream.emit("end");
  });

  const transformStream = createTransform();

  readStream.pipe(transformStream);

  return transformStream;
}

function createTransform() {
  const transformStream = new Transform({
    readableObjectMode: false, // Output should be Buffers or strings
    writableObjectMode: true, // Input will be objects,
    transform({ PayloadPart: { Bytes } = {} } = {}, encoding, callback) {
      const text = Buffer.from(Bytes)?.toString() ?? "";
      // Push the transformed data to the next handler in the stream
      this.push(text);
      callback();
    }
  });

  return transformStream;
}

[rjvgupta]

Hey Aaron, thanks for following up on this. I was able to try w/ v3 and can confirm, that it worked without needing to override the SDK with my changes. This is a reasonable workaround, but will take some work to modify the v2 code to v3 and think we should deploy the fix requested. Thank You, Rajiv Gupta Manager, Analytics Specialist Solutions Architects p: 949-296-5163 | e: ***@***.******@***.***> Thoughts on our interaction? Provide feedback here<https://feedback.aws.amazon.com/?ea=rjvgupta&fn=Rajiv&ln=Gupta>. From: Aaron V ***@***.***> Reply-To: aws/aws-sdk-js ***@***.***> Date: Thursday, March 21, 2024 at 7:43 PM To: aws/aws-sdk-js ***@***.***> Cc: "Gupta, Rajiv" ***@***.***>, Mention ***@***.***> Subject: Re: [aws/aws-sdk-js] ReferenceError in SageMaker invokeEndpointWithResponseStream (Issue #4580) @rjvgupta<https://github.com/rjvgupta> if you're only using this part of the AWS sdk, v3 of the sdk solves this problem, and consuming the output of it is a lot easier. e.g. const { Body } = await sageMakerRuntimeClient.send( new InvokeEndpointWithResponseStreamCommand(params) ); const readStream = Readable.from(Body); readStream.on("error", error => { console.error(`Sagemaker endpoint error: ${endpointName}`, error); readStream.emit("end"); }); const transformStream = createTransform(); readStream.pipe(transformStream); return transformStream; } function createTransform() { const transformStream = new Transform({ readableObjectMode: false, // Output should be Buffers or strings writableObjectMode: true, // Input will be objects, transform({ PayloadPart: { Bytes } = {} } = {}, encoding, callback) { const text = Buffer.from(Bytes)?.toString() ?? ""; // Push the transformed data to the next handler in the stream this.push(text); callback(); } }); return transformStream; } — Reply to this email directly, view it on GitHub<#4580 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALLHDRWXZEBLAR2POMIBT6TYZOLC5AVCNFSM6AAAAABCV2Z3CSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJUGIYTOMRRGA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[inf3rnus]

@rjvgupta np, and yeah, I agree. Ended up having to migrate to v3 myself because I ran into another problem where the deleteEndpoint method wouldn't actually delete endpoints, which means there's almost certainly other problems lurking.