is_at_end? throws errors

[kplimack]

1. there is a false negative test here, this is being called in a wrapper, but it doesnt detect that properly.
2. if this is supposed to force the handler to run, it shouldn't care about the run_list, it should be setting up a report handler, and certainly not barfing if I don't have it last in a run_list, the point of a handler is i dont care, it runs during start/finalize, it shouldnt break my run.

       Recipe: myOrg_inspec_handler::default
         * directory[/etc/chef/inspec-handler] action create (up to date)
         * directory[/etc/chef] action nothing (skipped due to action :nothing)
         * cookbook_file[/etc/chef/inspec_handler.rb] action nothing (skipped due to action :nothing)
         * chef_handler[myOrg::InspecHandler] action nothing (skipped due to action :nothing)
         * ruby_block[trigger_start_handlers] action nothing (skipped due to action :nothing)
         * directory[/etc/chef/inspec-handler/myOrg_inspec_handler] action create (up to date)
         * link[/etc/chef/inspec-handler/myOrg_inspec_handler/default.rb] action create (up to date)
         * inspec_handler_utils[inutil] action hard[2017-05-31T19:30:15+00:00] WARN: [Inspec Handler Utils] [/!\] inspec_handler::default is expected to be wrapped in myOrg_inspec_handler


           ================================================================================
           Error executing action `hard` on resource 'inspec_handler_utils[inutil]'
           ================================================================================

           RuntimeError
           ------------
           [Inspec Handler Utils] [--FATAL--]
           @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
           @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
           @ myOrg_inspec_handler is NOT placed at the end of your runlist. Make sure this cookbook runs last. @
           @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
           @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

           Cookbook Trace:
           ---------------
           /tmp/kitchen/cache/cookbooks/inspec_handler/libraries/inspec-handler-utils_provider.rb:90:in `is_at_end?'
           /tmp/kitchen/cache/cookbooks/inspec_handler/libraries/inspec-handler-utils_provider.rb:80:in `utils'
           /tmp/kitchen/cache/cookbooks/inspec_handler/libraries/inspec-handler-utils_provider.rb:67:in `action_hard'

[kplimack]

i seriously can't be expected to add this to the run_list of everything at the end, i should be able to add the hander via a base-role-cookbook (or whatever) and thats it.

[electriquo]

firstly, thank you for this cookbook, just need to retouch it a little bit so all of us could utilize it across the globe. the issue above can be resolved easily by implementing the same mechanism that appears in minitest-chef-handler. specifically, see this block, it should be something like...

if result && result.nonzero?
  ::Chef::Client.when_run_completes_successfully do |run_status|
    invoke_inspec_chef_handler
  end
end

i wish i could contribute some code, though i am sure you can crack it easily :)
(maybe i should open a separate issue for this one)

adding this functionality will always invoke the inspec execution at the end of the chef-client, regardless of the runlist order, which cannot be maintained when using chef, since writing the cookbook are constructed via composition pattern, which implies that imposing an order on the run list (having inspec at the end of the run list) is a against the composition pattern.

[sidxz]

@kplimack It has been written as an HWRP, and I did not find any good way to dynamically modify the chef run list during a chef-client run.
The inspec handler cookbook needs to be run last. If inspec test runs before convergence there will be false positives. (Though in the subsequent run there won't be any). We made this :
inspec_handler_utils do ensure_last true action :hard end
to rely less on an 'honor based' system and force developers to put this at the end.
For the time being, you can change the action to :soft. This would generate a warning and will not fail your client run. (However, there will be cases of false positives if it is not the last thing to be run. These false positive will vanish after the first run. i.e when a system differs the first run might report false positives which would be stabilized in the next run).

This is a workaround, and I am looking at @y0y0z suggestion to find a permanent solution.

@y0y0z Thank you for pointing this out.

[kplimack]

I wrote a new inspec handler from scratch that I'm going to publish as soon as my company permits me to open source it, hopefully this week. It addresses these issues among others, and is quite uncomplicated to grok through.

On Sun, Jun 4, 2017 at 7:40 AM Siddhant Rath ***@***.***> wrote: @kplimack <https://github.com/kplimack> It has been written as an HWRP, and I did not find any good way to dynamically modify the chef run list during a chef-client run. The inspec handler cookbook needs to be run last. If inspec test runs before convergence there will be false positives. (Though in the subsequent run there won't be any). We made this : inspec_handler_utils do ensure_last true action :hard end to rely less on an 'honor based' system and force developers to put this at the end. For the time being, you can change the action to :soft. This would generate a warning and will not fail your client run. (However, there will be cases of false positives if it is not the last thing to be run. These false positive will vanish after the first run. i.e when a system differs the first run might report false positives which would be stabilized in the next run). This is a workaround, and I am looking at @y0y0z <https://github.com/y0y0z> suggestion to find a permanent solution. @y0y0z <https://github.com/y0y0z> Thank you for pointing this out. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABFrJt4LyoX6PhdBm-jeA9q0h9VU1Hulks5sArPAgaJpZM4NsFqF> .

-- *Jake Plimack* *e. jake.plimack@gmail.com <jake.plimack@gmail.com>* *m. 917.838.7685*

[electriquo]

@kplimack contribution is not a bad word, but no need to create a different development branch -- in github, we call it a pull request :)
sometimes, a different development branch cannot be avoided (usually when there is a conflict of interest or the maintainer avoided the project).

you are welcome to do as you please, though give it a thought for a minute...

@sidxz thank you for looking into it

[kplimack]

My code is completely unique and does not share anything in common with yours. Sorry

On Sun, Jun 4, 2017 at 10:46 AM y0y0z ***@***.***> wrote: @kplimack <https://github.com/kplimack> contribution is not a bad word, but no need to create a different development branch -- in github, we call it pull request <https://help.github.com/articles/about-pull-requests/> :) sometimes, a different development branch cannot be avoided (usually when there is a conflict of interest or the maintainer avoided the project). you are welcome to do as you please, though give it a thought for a minute... @sidxz <https://github.com/sidxz> thank you for looking into it — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABFrJnMW77MezPRlV6p2MyggoSjzwHYnks5sAt9_gaJpZM4NsFqF> .

-- *Jake Plimack* *e. jake.plimack@gmail.com <jake.plimack@gmail.com>* *m. 917.838.7685*

[electriquo]

@sidxz also wanted to note that code in thelibraries and definitions directories are not getting into the chef runlist, and thus there is no ability to test them using chefspec, and each one that utilize this inspec-handler cookbook must always provide a mocking for the libraries code of inspec-handler, kinda cumbersome and can be avoided. if you split the libraries into /providers and /resources (see (custom resource)[https://docs.chef.io/custom_resources.html]) then you could easily provide a chefspec matcher.

from chefspec:

NOTE: If your cookbook exposes LWRPs, it is highly recommended you also create a libraries/matchers.rb file as outlined below in the "Packaging Custom Matchers" section. You should never step_into an LWRP unless you are testing it. Never step_into an LWRP from another cookbook!

check aws cookbook for an example of both splitting the libraries into light-weight resource provider (lwrp) and providing a chefspec matcher.

more of which, i had to patch the cookbook to introduce the following for it to work. i took minitest-chef-handler as a reference which supports out of the box in all of the following:

1. install inspec rubygem to the embedded ruby in chef
2. dynamically create a whitelist for inspec_handler as it does observe the correct expended run list to load the corresponding tests
3. store node attributes, load (merge node attribute with kitchen dna file) and expose them before running inspec examples, as there is no way to access the node attribute

[sidxz]

@y0y0z Completely agree to the fact of splitting it. I will update it in the next release.

"store node attributes, load (merge node attribute with kitchen dna file) and expose them before running inspec examples, as there is no way to access the node attribute"
The spirit in which this was built was to include the inspec tests in their respective cookbooks as templates. These templates generate the inspect test and put them in the respective location ( say '/etc/chef/inspec-handler//recipename'). Since these are templates they have complete access to the resolved attributes.
Have a look at this generator cookbook that creates the structure for you https://github.com/sidxz/chef-code-generator
Basically, with each recipe that is created the generator cookbook:

• Creates a corresponding template at templates/default/inspec-tests/<recpie_name>_inspec.erb (This is where you put your inspec test)
• In the beginning of the recipe, it adds this code to copy the template to the desired location.

# Generate InSpec Test Files
# These tests are used in the chef automate delivery pipeline.
# Create test dir
directory "/etc/inspec-tests/coe-freeradius" do
recursive true
end
# Create test file
template "/etc/inspec-tests/coe-freeradius/default.rb" do
source "inspec-tests/default_inspec.erb"
owner 'root'
group 'root'
mode '750'
end
# <----------  Start writing recipe from here ------->

You can pass in variables as you do to templates and generate the inspec test dynamically when the cookbook is converged.

Correct me if I have misunderstood you.

[electriquo]

@sidxz why won't you take a closer look at the minitest-chef-handler cookbook (i know i mentioned this cookbook few times, i have no affiliation with this cookbook. it just works fine), which follows the idea of convention over configuration, holds the test in files/default/test/*_test.rb, where * is a cookbook name (having the test in templates is kinda losing the semantics of chef directory names\structure) , and does not add any code blocks. reminding you that code blocks must always be maintained and tested and i think you'd like to avoid it.