Scraping from Arabic news

[RabeaAffan24]

Next steps:

• Get all archive (+-300 pages)
• generate JSON (JSON serialisation)
• integrate Panet scraping into Anyway ETL

[atalyaalon]

@BusinessLanguage looks good!
However right now this code writes into a local file. Not sure we want to merge this in that way.
We can merge just to make sure code is in our repo - however it's not a code that will run in prod.
@ziv17 @shaysw any thoughts?

[ziv17]

Very nice!

Do we want to use these accidents for our reports and infographics?
If yes, then we need to add them to our database. To do this, I think we need:

• coordinate the values of entities in the results file to those that are used in our database (e.g. injury severity, street name, accident severity etc.) Currently for these entities we use codes(numbers), and English names in the code, and have translation to Hebrew using pybabel.
• Then add the data (accidents, injured, etc.) to our database.

[ziv17]

Hi, I would like us to use type hints in function parameters and variables.

[ziv17]

Hi,

• In python we use lowercase and underscore for variables and function/method names.
• I prefer not to change the variable loop in the loop. It is better to use a different variable.

[ziv17]

Are we OK that the API key is in our code, in a public repo?

[RabeaAffan24]

Still working on this one..
Accidently pushed it to the PR

[ziv17]

Hi, the style we use for file names is lowercase with underscore between words.

[ziv17]

Hi, Well done!
See some technical comments below.
How is this code going to be incorporated in our application? I think it worth a discussion.

[RabeaAffan24]

@ziv17 Thank you for your comments. Will amend those issues soon.

regarding your question, incorporating the obtained data in the database will be carried out after the newsflash been translated (using Google API) and then will undergo the same process as your mainstream data.