Problem submitting a CSV as a google doc

[sebbacon]

Someone is using a CSV which is a google doc to submit their return (link in my email) and is getting the folowing error:

2019-08-04T11:43:42.455652+00:00 app[web.1]: [2019-08-04 11:43:42,453] ERROR in app: Exception on /19A2/preview [GET]
2019-08-04T11:43:42.455672+00:00 app[web.1]: Traceback (most recent call last):
2019-08-04T11:43:42.455674+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
2019-08-04T11:43:42.455676+00:00 app[web.1]: response = self.full_dispatch_request()
2019-08-04T11:43:42.455678+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
2019-08-04T11:43:42.455679+00:00 app[web.1]: rv = self.handle_user_exception(e)
2019-08-04T11:43:42.455680+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
2019-08-04T11:43:42.455682+00:00 app[web.1]: reraise(exc_type, exc_value, tb)
2019-08-04T11:43:42.455683+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
2019-08-04T11:43:42.455685+00:00 app[web.1]: raise value
2019-08-04T11:43:42.455686+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
2019-08-04T11:43:42.455688+00:00 app[web.1]: rv = self.dispatch_request()
2019-08-04T11:43:42.455689+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
2019-08-04T11:43:42.455691+00:00 app[web.1]: return self.view_functions[rule.endpoint](**req.view_args)
2019-08-04T11:43:42.455692+00:00 app[web.1]: File "/app/vatreturn.py", line 42, in decorated_function
2019-08-04T11:43:42.455694+00:00 app[web.1]: return f(*args, **kwargs)
2019-08-04T11:43:42.455695+00:00 app[web.1]: File "/app/vatreturn.py", line 183, in preview_return
2019-08-04T11:43:42.455696+00:00 app[web.1]: g.data = return_data(g.period_key, g.period_end, g.vat_csv)
2019-08-04T11:43:42.455698+00:00 app[web.1]: File "/app/vatreturn.py", line 143, in return_data
2019-08-04T11:43:42.455699+00:00 app[web.1]: df = pd.read_csv(vat_csv)
2019-08-04T11:43:42.455700+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/pandas/io/parsers.py", line 702, in parser_f
2019-08-04T11:43:42.455702+00:00 app[web.1]: return _read(filepath_or_buffer, kwds)
2019-08-04T11:43:42.455703+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/pandas/io/parsers.py", line 435, in _read
2019-08-04T11:43:42.455704+00:00 app[web.1]: data = parser.read(nrows)
2019-08-04T11:43:42.455705+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/pandas/io/parsers.py", line 1139, in read
2019-08-04T11:43:42.455707+00:00 app[web.1]: ret = self._engine.read(nrows)
2019-08-04T11:43:42.455708+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/pandas/io/parsers.py", line 1995, in read
2019-08-04T11:43:42.455709+00:00 app[web.1]: data = self._reader.read(nrows)
2019-08-04T11:43:42.455710+00:00 app[web.1]: File "pandas/_libs/parsers.pyx", line 899, in pandas._libs.parsers.TextReader.read
2019-08-04T11:43:42.455712+00:00 app[web.1]: File "pandas/_libs/parsers.pyx", line 914, in pandas._libs.parsers.TextReader._read_low_memory
2019-08-04T11:43:42.455713+00:00 app[web.1]: File "pandas/_libs/parsers.pyx", line 968, in pandas._libs.parsers.TextReader._read_rows
2019-08-04T11:43:42.455715+00:00 app[web.1]: File "pandas/_libs/parsers.pyx", line 955, in pandas._libs.parsers.TextReader._tokenize_rows
2019-08-04T11:43:42.455716+00:00 app[web.1]: File "pandas/_libs/parsers.pyx", line 2172, in pandas._libs.parsers.raise_parser_error
2019-08-04T11:43:42.455717+00:00 app[web.1]: pandas.errors.ParserError: Error tokenizing data. C error: Expected 1 fields in line 6, saw 2

[pubmania]

I think I have managed to recreate this issue - This happens when I am providing the CSV path from nextcloud created and saved using Microsoft Excel.

2019-09-05T12:04:15.847671+00:00 app[web.1]:   File "pandas/_libs/parsers.pyx", line 881, in pandas._libs.parsers.TextReader.read
2019-09-05T12:04:15.847672+00:00 app[web.1]:   File "pandas/_libs/parsers.pyx", line 896, in pandas._libs.parsers.TextReader._read_low_memory
2019-09-05T12:04:15.847673+00:00 app[web.1]:   File "pandas/_libs/parsers.pyx", line 950, in pandas._libs.parsers.TextReader._read_rows
2019-09-05T12:04:15.847675+00:00 app[web.1]:   File "pandas/_libs/parsers.pyx", line 937, in pandas._libs.parsers.TextReader._tokenize_rows
2019-09-05T12:04:15.847676+00:00 app[web.1]:   File "pandas/_libs/parsers.pyx", line 2132, in pandas._libs.parsers.raise_parser_error
2019-09-05T12:04:15.847677+00:00 app[web.1]: pandas.errors.ParserError: Error tokenizing data. C error: Expected 1 fields in line 8, saw 3

[sebbacon]

So the problem in the first case was that the person was providing a URL to a google doc which was a CSV. I.e. it wasn't actually a CSV.

The trick was to put it in a Google Sheet and publish the worksheet as a CSV.

I can help you check the URL if you want to upload some dummy data and then share the URL?

[pubmania]

Hi Seb - I am anyway using dummy data. I will share the Nextcloud link through email - will come to your gmail account. The csv file I used is definitely CSV and infact I downloaded it from the googlesheet published CSV with no changes whatsoever. Will share it shortly.

Once again many thanks for this tool. :)

[pubmania]

ok I have just shared the link to the CSV file.... although You will not be able to use it from your hosted solution because I have modified the code to accept other columns. Perhaps I should create a CSV file that works with FRS for your set-up.

I have created one more csv with three columns so you can test.

[sebbacon]

So to be clear, you're providing the CSV link itself, not the URL of the page that shows a preview of the CSV?

[sebbacon]

You should also note you won't be able to use your own version to actually submit a return until you submit it and get it approved by HMRC (I need to mention this in the docs).

No reason why they wouldn't (they accepted my version after all) but it can take a while.

[pubmania]

yeah I have just received an email where they have asked to book a slot for providing demo - I have gone ahead and requested it. As a backup for this period I have asked my accountant to use some bridging software and get it submitted but for next time I would use this if they approve it. :)

Thanks for the explanation - I will give it a shot with the "download" link. Please can I request you to edit the image in your response so the Nextcloud URL is not visible. Rest I guess might be useful for other users too.

[pubmania]

ok tested with the link on "Download" and then it works fine. This can be closed. By the way I have added a lot of information to the readme.md file on my fork which may be useful for anyone trying to deploy on their own - Feel free to add it here if you see fit.

[sebbacon]

Thanks. Could you make a PR? Perhaps separating out the mysql stuff to a different file or something?

[pubmania]

Two problems here with PR - I don't quite know how to do it so I started trying but if I understood correctly PR will merge all changes on my fork to yours atleast for whole of the Readme.md if those changes are accepted. Now I have made some changes specific to the additional fields that are required for my fork to work - for instance the table structure - instead of 3 columns it is 8 columns. I could perhaps update it to show both versions but it might make it confusing for the end user.

I then thought, ok I will create another fork and just copy paste stuff relevant to your original work but github wont let me make another fork. Your work is so nice and helpful that I want to be able to help somehow but my skills (or lack of it) lets me down.

Will it help if I send an email with updated readme and associated images?

[sebbacon]

Ha, fair enough!

I've not looked over your code but it seems to me that supporting 8 columns rather than 3 would be useful for anyone? In which case it would potentially be fine to merge the whole lot (except the stuff about mysql).

I'd be happy to make a PR myself based off your code, it's just I don't really have the time. If you just want to leave it then maybe I'll come back to it; if you want to try to make a PR I could try to help out.

[pubmania]

Ah the mysql part is quite independent of this tool really. All it does is instead of pulling numbers from another sheet on to a pivot like you are doing, it pulls the data from a database server where my invoice and expenses are maintained. Part of your code that I have modified is quite literally this - #3 (comment)

Then when I pushed the build to Heroku, I figured I will have to update landing page where I have agreed with you 100% :) and also the image for spreadsheet and all other images in the readme doc.

I will try and create a PR so once you get time you can compare stuff decide whether to merge the stuff or not.