CreateProcessW(): Invalid access to memory location

[docwhat]

I'm getting these errors randomly (and hard to reproduce) on my boxen:

Chef::Exceptions::MultipleFailures: Multiple failures occurred:
* SystemCallError occurred in chef run: powershell_script[ssl-cert: permissions] (bigdatalab-role-windows::_ssl_cert line 47) had an error: SystemCallError: Invalid access to memory location. - CreateProcessW
* SystemCallError occurred in delayed notification: windows_task[\chef-client] (chef-client::task line 32) had an error: SystemCallError: Invalid access to memory location. - CreateProcessW


C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/mixlib-shellout-2.1.0-universal-mingw32/lib/mixlib/shellout/windows/core_ext.rb:346:in `create'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/mixlib-shellout-2.1.0-universal-mingw32/lib/mixlib/shellout/windows.rb:86:in `run_command'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/mixlib-shellout-2.1.0-universal-mingw32/lib/mixlib/shellout.rb:259:in `run_command'
C:/opscode/chef/embedded/apps/chef/lib/chef/mixin/shell_out.rb:97:in `shell_out_command'
C:/opscode/chef/embedded/apps/chef/lib/chef/mixin/shell_out.rb:50:in `shell_out'
C:/opscode/chef/embedded/apps/chef/lib/chef/mixin/shell_out.rb:55:in `shell_out!'
C:/opscode/chef/embedded/apps/chef/lib/chef/provider/powershell_script.rb:79:in `block in validate_script_syntax!'
C:/opscode/chef/embedded/lib/ruby/2.0.0/tempfile.rb:324:in `open'
C:/opscode/chef/embedded/apps/chef/lib/chef/provider/powershell_script.rb:64:in `validate_script_syntax!'
C:/opscode/chef/embedded/apps/chef/lib/chef/provider/powershell_script.rb:33:in `action_run'
C:/opscode/chef/embedded/apps/chef/lib/chef/provider.rb:140:in `run_action'
C:/opscode/chef/embedded/apps/chef/lib/chef/resource.rb:584:in `run_action'
C:/opscode/chef/embedded/apps/chef/lib/chef/runner.rb:49:in `run_action'
C:/opscode/chef/embedded/apps/chef/lib/chef/runner.rb:81:in `block (2 levels) in converge'
C:/opscode/chef/embedded/apps/chef/lib/chef/runner.rb:81:in `each'
C:/opscode/chef/embedded/apps/chef/lib/chef/runner.rb:81:in `block in converge'
C:/opscode/chef/embedded/apps/chef/lib/chef/resource_collection/resource_list.rb:83:in `block in execute_each_resource'
C:/opscode/chef/embedded/apps/chef/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
C:/opscode/chef/embedded/apps/chef/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
C:/opscode/chef/embedded/apps/chef/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
C:/opscode/chef/embedded/apps/chef/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
C:/opscode/chef/embedded/apps/chef/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
C:/opscode/chef/embedded/apps/chef/lib/chef/resource_collection/resource_list.rb:81:in `execute_each_resource'
C:/opscode/chef/embedded/apps/chef/lib/chef/runner.rb:80:in `converge'
C:/opscode/chef/embedded/apps/chef/lib/chef/client.rb:654:in `block in converge'
C:/opscode/chef/embedded/apps/chef/lib/chef/client.rb:649:in `catch'
C:/opscode/chef/embedded/apps/chef/lib/chef/client.rb:649:in `converge'
C:/opscode/chef/embedded/apps/chef/lib/chef/client.rb:688:in `converge_and_save'
C:/opscode/chef/embedded/apps/chef/lib/chef/client.rb:269:in `run'
C:/opscode/chef/embedded/apps/chef/lib/chef/application.rb:252:in `run_with_graceful_exit_option'
C:/opscode/chef/embedded/apps/chef/lib/chef/application.rb:229:in `block in run_chef_client'
C:/opscode/chef/embedded/apps/chef/lib/chef/local_mode.rb:39:in `with_server_connectivity'
C:/opscode/chef/embedded/apps/chef/lib/chef/application.rb:212:in `run_chef_client'
C:/opscode/chef/embedded/apps/chef/lib/chef/application/client.rb:375:in `run_application'
C:/opscode/chef/embedded/apps/chef/lib/chef/application.rb:60:in `run'
C:/opscode/chef/embedded/apps/chef/bin/chef-client:26:in `<top (required)>'
C:/opscode/chef/bin/chef-client:67:in `load'
C:/opscode/chef/bin/chef-client:67:in `<main>'

They don't happen at any particular recipe for any particular cookbook. That is, they are pretty random.

What I think is happening is that CreateProcessW() is passing in a "constant" version of cmd. According to the MSDN Documentation's for lpCommandLine:

The Unicode version of this function, CreateProcessW, can modify the contents of this string. Therefore, this parameter cannot be a pointer to read-only memory (such as a const variable or a literal string). If this parameter is a constant string, the function may cause an access violation.

The error message "Invalid access to memory location" is consistent with the second argument (lpCommandLine) being within read-only memory.

I think the fix is to change core_ext.rb line 226 to use an allocated bit of memory:

    if hash['command_line']
      cmd = FFI::MemoryPointer.new(:string, 1024)
      cmd.write_string hash['command_line'].to_wide_string
    end

I'm not sure 1024 is the right value. Other examples I have seen use things like (reverting to C/C++) the following:

WCHAR lpCommandLine[MAX_PATH * 2];
/* Make a copy of cmd so CreateProcessW() can modify it if it needs to. */
_tcscpy_s(lpCommandLine, MAX_PATH *2, (WCHAR *)cmd);

or

// http://stackoverflow.com/questions/4514027/createprocessw-acess-violation-in-rtlinitunicodestring

//I'm copying the string here because CreateProcessW mutates its arguments
wchar_t *lpCommandLine = _wcsdup(cmd);
...
free(lpCommandLine);

If it helps, there is a Microsoft blog post about why CreateProccesW() has historically had this behavior: http://blogs.msdn.com/b/oldnewthing/archive/2009/06/01/9673254.aspx

[pburkholder]

@adamedx @ksubrama @smurawski - Could one of our Windows gurus give this a look? Thanks, Peter

[ksubrama]

@docwhat Could you give us the powershell block/command being used that trips this? I don't believe that the string being provided is read-only. It is composed dynamically by the powershell_script resource and I don't see any big changes to mixlib-shellout that would cause this to not work when it would have been working before. If it is the read-only argument issue, having a repro would be nice to help track it down.

[docwhat]

We have about 50 Windows VMs and we get about one error a day. So I don't have any reproducible string.

Since I use the chef-client::task recipe, it is one of the ones that trip this bug. So you could use that.

I think that it mainly happens in :delayed resources.

[docwhat]

To further clarify. I also tried combinations of cookbooks in a vagrant and could never reproduce it either. No matter how I tried.

If I was going to go further I'd try using Windows debugging tools in a Vagrant, but I'm not familiar with debugging at that level in Windows nor at that level with FFI.

Is there a way I can do some kind of introspection inside the FFI code that I could run on one or more of the live systems?

I can't recompile anything on them that chef_gem can't do; they are production. But I'd be willing to monkey patch them.

They are all running 12.4.1. I could try 12.4.3 I guess.

I don't recall which version of Chef was on them previously and I couldn't tell you for sure if it was an upgrade the Chef, cookbooks, or Windows itself that made this likely to happen.

[ksubrama]

@docwhat The upstream library we use was apparently binding the second argument as an input only buffer. I'm unsure as the what the actual effect of that is - the FFI documentation claims that it is a hint that they can use for "optimizations". But it's a possible lead.

[docwhat]

The MSDN docs said it must be both input and output. I imagine the optimization include putting the variable in read only memory.

Sounds like a significant lead!

Pointers to the code for those of us playing along at home?

[ksubrama]

https://github.com/djberg96/win32-process/blob/ffi/lib/win32/process/functions.rb#L59 . The github issue I referenced above also mentions other possible bugs in the code.

[vanessazou]

Is there any solution for this issue? We have the same issue and could not find and solution.
Thank you