fp importer seems to complain if plot already exists

[ashenkin]

looking at the fp_importer class, this doesn't seem like it should be an issue. But, after I blew away the IVI-01 plot for Carl (see the sql in db/misc_sql), he's getting errors trying to upload again:

[14:51:59] [WARN] Skipping item which already exists on line 2 - No permission to override Plot from a different batch: {"id":196,"site_id":null,"plot_code":"IVI-01","plot_desc":"GEM Ivindo Plot 1","latitude":null,"longitude":null,"ne_latitude":null,"ne_longitude":null,"nw_latitude":null,"nw_longitude":null,"se_latitude":null,"se_longitude":null,"sw_latitude":null,"sw_longitude":null,"plot_shape":null,"plot_area_m2":null,"elevation_m":null,"slope_deg":null,"aspect_deg":null,"start_date_rainfor":null,"start_date_ccycle":null,"end_date_ccycle":null,"data_collected_code":null,"data_collection_list":null,"small_stem_plot_area":null,"cwd_transect_area_m2":null,"partitionning_collars_installation_date":null,"disturbances":null,"created_at":"2016-06-02T10:41:50.968Z","updated_at":"2016-06-02T10:41:50.972Z","fp_id":3405,"batch_id":271}
[14:51:59] [WARN] Skipping item which already exists on line 3 - No permission to override Plot from a different batch: {"id":196,"site_id":null,"plot_code":"IVI-01","plot_desc":"GEM Ivindo Plot 1","latitude":null,"longitude":null,"ne_latitude":null,"ne_longitude":null,"nw_latitude":null,"nw_longitude":null,"se_latitude":null,"se_longitude":null,"sw_latitude":null,"sw_longitude":null,"plot_shape":null,"plot_area_m2":null,"elevation_m":null,"slope_deg":null,"aspect_deg":null,"start_date_rainfor":null,"start_date_ccycle":null,"end_date_ccycle":null,"data_collected_code":null,"data_collection_list":null,"small_stem_plot_area":null,"cwd_transect_area_m2":null,"partitionning_collars_installation_date":null,"disturbances":null,"created_at":"2016-06-02T10:41:50.968Z","updated_at":"2016-06-02T10:41:50.972Z","fp_id":3405,"batch_id":271}
[14:51:59] [WARN] Skipping item which already exists on line 4 - No permission to override Plot from a different batch: {"id":196,"site_id":null,"plot_code":"IVI-01","plot_desc":"GEM Ivindo Plot 1","latitude":null,"longitude":null,"ne_latitude":null,"ne_longitude":null,"nw_latitude":null,"nw_longitude":null,"se_latitude":null,"se_longitude":null,"sw_latitude":null,"sw_longitude":null,"plot_shape":null,"plot_area_m2":null,"elevation_m":null,"slope_deg":null,"aspect_deg":null,"start_date_rainfor":null,"start_date_ccycle":null,"end_date_ccycle":null,"data_collected_code":null,"data_collection_list":null,"small_stem_plot_area":null,"cwd_transect_area_m2":null,"partitionning_collars_installation_date":null,"disturbances":null,"created_at":"2016-06-02T10:41:50.968Z","updated_at":"2016-06-02T10:41:50.972Z","fp_id":3405,"batch_id":271}

[mankind]

Fix for issue #47. The problem is triggered in the row_importer class by this line of code object.can_overwrite(@batch_id, @overwrite_batch_id) when we call attempt_to_overwrite!(object) method, Instead of using object.batch_id, it is using batch_id which results in an attempt to create a new plot in the database using the records of a plot that already exists. That in turn causes the reported error: No permission to override Plot a different batch

The way the code is currently designed, it will always create a new batch in the database each time an upload is submitted irrespective of whether it is a new upload or simply an update which was the case with Allie. I didn't want to change that, so I added a new boolean column named duplicate to the batches table. This will allow us to easily delete duplicate batch record that was created during update but are redundant. In order to fix the reported issue, I had to reuse existing batch whenever an import is an update as Allie was doing, So the boolean duplicate column allows us to set any new batch record* created during an up date to true and later we can just delete all batch record where the duplicate value is equal to true.

Ed

[ashenkin]

Thanks for this Ben. Yes, this makes sense. I can't actually blow away the corresponding record in the 'plots' table, as there are other existing data tied to that plot. In general, updating a forestplots import is a royal pain, but also important to be able to do.

So as I understand it, your fix will add a duplicate batch record so we can maintain traceability while allowing the update... Have you tested this yet? If not, I should probably run the migration on my dev machine before doing so on the production server, right?

[mankind]

Allie the fix is from me Edward not Ben. We spoke on phone recently, followed by email exchange.

I have tested the fix on my dev machine and it works. But still test it for yourself on your dev machine before deploying it to the production server. Just upload a forest import as usual, then run your sql to delete the tree and dbh_measurement, then upload the forest import again and it will work.

It is not my fix that adds a duplicate batch record. That is how the code always worked even before I touched the code. The previous developers designed it that way. That is during upload or import to create a new batch record. My fix just adds a way to remove the duplicate batch later.

This fix preserves the record in the plots table and records inserted into other tables like subplot, fp_family etc.

The only modification I made to the forest import csv was to ensure that sub_plot_code was not empty. If sub_plot_code is not available, then a duplicate subplot record is created. Again this is how I met the code. To avoid that just ensure sub_plot_code is available in the csv for upload.

After running it on your dev machine, let me know if you run into any issues.