Skip to content

Ld Json Schema Scraper#50

Open
shaniqwa wants to merge 102 commits intojadkins89:masterfrom
shaniqwa:master
Open

Ld Json Schema Scraper#50
shaniqwa wants to merge 102 commits intojadkins89:masterfrom
shaniqwa:master

Conversation

@shaniqwa
Copy link
Contributor

@shaniqwa shaniqwa commented Aug 15, 2021 •
edited
Loading

adding support for a huge variety of recipe websites worldwide, this scraper is looking for the google recipe JSON schema and populates the recipe object.

Happy to hear your thoughts and add improvements
Cheers, Shani

shaniqwa added 30 commits April 28, 2021 15:06
@shaniqwa
support jamie oliver scraping
6fdf15a
@shaniqwa
fix some tests. add defaultSetDescription to BaseScraper
ecbe2bd
@shaniqwa
fix more tests
cda11c4
@shaniqwa
rename
8669a34
@shaniqwa
fix SimplyRecipes tests
e451674
@shaniqwa
fix yummly. update puppeteer to latest version
a005ae9
@shaniqwa
fix thespruceeats tests
b39a9f6
@shaniqwa
theSpruceEast - add recipe description
4db4c1d
@shaniqwa
fix woolworths
55d8d9f
@shaniqwa
closetCooking add description
27e38f9
@shaniqwa
try to change travis node version
00ba239
@shaniqwa
revert travis node version to stable
6b217c2 
add logs
@shaniqwa
more logs
caa38c9
@shaniqwa
logs
baf8a87
@shaniqwa
log
f6ff471
@shaniqwa
log
affc05a
@shaniqwa
log
70314e3
@shaniqwa
skip should fetch the expected recipe test if server is not responding
f04e3a4
@shaniqwa
add ambitiouskitchen description
20583c3
@shaniqwa
add recipetineats description
4736af1
@shaniqwa
add GimmeDelicious description
b47e379
@shaniqwa
add tastOfHome description
1e07117
@shaniqwa
add SeriousEats description
71e1029
@shaniqwa
add therecipecritic description
7045020
@shaniqwa
add myrecipes description
f426a0f
@shaniqwa
bonappetit add descriptions and tags
7a7e1ae
@shaniqwa
tastesbetterfromscratch add description and tags
5821206
@shaniqwa
kitchen stories - add description and tags
dd2f66a
@shaniqwa
foodandwine add description
972b953
@shaniqwa
NomNomPaleo add description
60c7803
shaniqwa added 30 commits August 9, 2021 17:50
@shaniqwa
woolworths should use the ld json schema, but it's still not working …
e651c01 
…since server response is 403
@shaniqwa
remove woolworths test
d776a87
@shaniqwa
fix handling unsupported / invalid urls
f4533ac
@shaniqwa
html decode - add regex for removing specific short cods
fbad66d 
fix tests: mels kitchen now using ld json scraper
add expected results to all ld json scraper test cases
fix bug/edge case handling in description and servings
@shaniqwa
fix test
ee6379c
@shaniqwa
Merge pull request #2 from shaniqwa/JSON+LD
daa4b0a 
add Json+ld scraper
@shaniqwa
add support for HowToSection. add test for a website requested in iss…
ba11390 
…ue (pav-bhaji-recipe-mumbai the longest recipe ever!!)
@shaniqwa
Merge pull request #3 from shaniqwa/JSON+LD
2ca3b61 
add support for HowToSection. add test for a website requested in iss…
@shaniqwa
fix typo
00ae10b
@shaniqwa
add logs
36be742
@shaniqwa
Throw errors
7485a33
@shaniqwa
update puppeteer to latest version
bd18d28
@shaniqwa
don't use puppeteer in json scraper
89e26fb
@shaniqwa
remove logs, fix test
1a85858
@shaniqwa
fix tests
d89b6d9
@shaniqwa
Merge pull request #4 from shaniqwa/JSON+LD
48c8413 
Json+ld
@shaniqwa
if no recipe schema was found, do not throw error. return page title,…
d53f5cc 
… image & description meta data
@shaniqwa
Merge branch 'master' into JSON+LD
97afad1 
* master:
@shaniqwa
fix all tests
bf64b78
@shaniqwa
fix 404 handling
0aac237
@shaniqwa
fix test. make expected test to match 80%. more scrapers now use defa…
caa800f 
…ult ld json
@shaniqwa
more fixes
5e7223e
@shaniqwa
change to a proper module. update dependencies
e0c1e34
@shaniqwa
remove puppeteer
28ddf8a
@shaniqwa
fix main path
8e2de13
@shaniqwa
fix files
72286a3
@shaniqwa
fix file path import
fc39ded
@shaniqwa
fix
c357254
@shaniqwa
remove console.logs
d6e7250
@shaniqwa
bump version
d4acc04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@shaniqwa