Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrating data to newUserFunctionId tables as part of Consumption plan support #1071

Merged
merged 12 commits into from
May 29, 2024

Conversation

MaddyDev
Copy link
Contributor

@MaddyDev MaddyDev commented Apr 25, 2024

New UserFunctionId:
We have used the WEBSITE_SITE_NAME instead of the host id for creating the new userFunctionId

Migration steps followed:

  1. While creating the lease table entry in the global state table, check if there a value with the oldUserFunctionId and use that value while inserting a new entry with the new userFunctionId to the global state table. And delete the oldUserFunctionId entry.
  2. Create new leases table with new hash and if old leases table exists, copy over all the data from it and delete the table.

Test: To Be Done

@AmeyaRele AmeyaRele requested a review from alrod May 2, 2024 03:49
src/SqlBindingConstants.cs Outdated Show resolved Hide resolved
src/TriggerBinding/SqlTriggerListener.cs Outdated Show resolved Hide resolved
@@ -54,6 +54,15 @@ public static string GetConnectionString(string connectionStringSetting, IConfig
return connectionString;
}

public static string GetWebSiteName(IConfiguration configuration)
{
if (configuration == null)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens for local development? Is website site name set?

Copy link
Contributor Author

@MaddyDev MaddyDev May 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WEBSITE_SITE_NAME value is null for local development, it creates the hex value for userfunctionId only using the functionName for local development

src/Telemetry/Telemetry.cs Outdated Show resolved Hide resolved
@@ -78,8 +78,9 @@ public async Task<IListener> CreateListenerAsync(ListenerFactoryContext context)
{
_ = context ?? throw new ArgumentNullException(nameof(context), "Missing listener context");

string userFunctionId = await this.GetUserFunctionIdAsync();
return new SqlTriggerListener<T>(this._connectionString, this._tableName, this._leasesTableName, userFunctionId, context.Executor, this._sqlOptions, this._logger, this._configuration);
string userFunctionId = this.GetUserFunctionId();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even after a migration is done once, are we still calculating these old + new values? Is that really needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we decided to have the queries such that if old values are present, migrate them and clean up else continue with new values.

src/TriggerBinding/SqlTriggerListener.cs Show resolved Hide resolved
BEGIN
-- Migrate LastSyncVersion from oldUserFunctionId if it exists and delete the record
Declare @lastSyncVersion bigint;
Select @lastSyncVersion = LastSyncVersion from az_func.GlobalState where UserFunctionID = '{this._oldUserFunctionId}' AND UserTableID = {userTableId}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plz check for sql injection possibilities

@chlafreniere
Copy link
Contributor

@Charles-Gagnon can you take another look?

Copy link
Contributor

@Charles-Gagnon Charles-Gagnon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's one potential bug to look at, and then please make sure to respond to Chris' comments as well. Otherwise looks good though!

Copy link
Contributor

@Charles-Gagnon Charles-Gagnon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would definitely recommend updating the README/MS docs to make it clear how the app name relates to the table and the consequences of changing that.

One thing I'm not sure we tested before is whether changing the app name previously would cause the host ID to change (and thus "lose" the history). I believe that would have been the case even previously, but now it's much more explicit so it'd be good to add that information in.

@MaddyDev
Copy link
Contributor Author

I would definitely recommend updating the README/MS docs to make it clear how the app name relates to the table and the consequences of changing that.

One thing I'm not sure we tested before is whether changing the app name previously would cause the host ID to change (and thus "lose" the history). I believe that would have been the case even previously, but now it's much more explicit so it'd be good to add that information in.

Will be adding the docs along with an end to end test in a separate PR. Thanks for calling this out.

@MaddyDev MaddyDev merged commit 160d9d6 into main May 29, 2024
2 checks passed
@MaddyDev MaddyDev deleted the maddy/newUserFunctionId branch May 29, 2024 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants